Systems and Methods for Adaptively Presenting a Keyboard on a Touch-Sensitive Display

ABSTRACT

Systems and methods for adaptively presenting a keyboard on a touch-sensitive display are disclosed herein. In one aspect, a method includes: monitoring typing inputs received from a user at a touch-sensitive display of an electronic device. The method also includes: determining whether the typing inputs are converging towards a midpoint of the touch-sensitive display or diverging away from the midpoint of the touch-sensitive display. In accordance with a determination that the typing inputs are converging towards the midpoint of the touch-sensitive display, the method includes: providing a first feedback to the user to indicate that the typing inputs are converging. In accordance with a determination that the typing inputs are diverging away from the midpoint of the touch-sensitive display, the method includes: providing a second feedback to the user to indicate that the typing inputs are diverging.

RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.16/273,025, filed Feb. 11, 2019, which is a divisional of U.S.application Ser. No. 15/199,672, filed Jun. 30, 2016 (the “ParentApplication”), now U.S. Pat. No. 10,203,873. The Parent Application is acontinuation-in-part of U.S. application Ser. No. 14/732,594, filed Jun.5, 2015, now U.S. Pat. No. 10,126,942 (the “First GrandparentApplication”).

The First Grandparent Application is also a continuation-in-part of U.S.patent application Ser. No. 13/442,855, filed Apr. 10, 2012, now U.S.Pat. No. 9,454,270, which is a continuation-in-part of U.S. patentapplication Ser. No. 12/234,053, filed Sep. 19, 2008, now U.S. Pat. No.8,325,141, which claims priority to U.S. Provisional Application No.60/973,691, filed Sep. 19, 2007.

The First Grandparent Application is also a continuation-in-part of U.S.application Ser. No. 13/308,428, filed Nov. 30, 2011, which is acontinuation-in-part of U.S. application Ser. No. 13/171,124, filed Jun.28, 2011, which claims priority to U.S. Provisional Application No.61/359,235, filed Jun. 28, 2010. U.S. application Ser. No. 13/308,428 isalso a continuation-in-part of U.S. application Ser. No. 12/234,053,filed Sep. 19, 2008, now U.S. Pat. No. 8,325,141, which claims priorityto Provisional Application Ser. No. 60/973,691, filed Sep. 19, 2007.

The First Grandparent Application is also a continuation-in-part of U.S.patent application Ser. No. 13/308,416, filed Nov. 30, 2011, now U.S.Pat. No. 9,110,590, which is a continuation-in-part of U.S. patentapplication Ser. No. 13/171,124, filed Jun. 28, 2011, which claimspriority to Provisional Application No. 61/359,235, filed Jun. 28, 2010.U.S. patent application Ser. No. 13/308,416 also claims priority toProvisional Application No. 61/472,799, filed Apr. 7, 2011, and toProvisional Application No. 61/418,279, filed Nov. 30, 2010, and is alsoa continuation-in-part of U.S. application Ser. No. 12/234,053, filedSep. 19, 2008, now U.S. Pat. No. 8,325,141, which claims priority toProvisional Application Ser. No. 60/973,691, filed Sep. 19, 2007.

The Parent Application is also a continuation-in-part of U.S. patentapplication Ser. No. 14/265,340, filed Apr. 29, 2014, now U.S. Pat. No.9,489,086 (the “Second Grandparent Application”), which claims priorityto U.S. Provisional Application No. 61/817,276, filed Apr. 29, 2013.

Each of the applications identified in the paragraphs above is herebyincorporated by reference in its respective entirety.

TECHNICAL FIELD

The disclosure herein relates to typing at a touch-sensitive displayand, in particular, to improved typing accuracy and input disambiguationby adaptively presenting a keyboard on a touch-sensitive display.

BACKGROUND

The origin of the modern keyboard as the primary method for inputtingtext and data from a human to a machine dates back to early typewritersin the 19th century. As computers were developed, it was a naturalevolution to adapt the typewriter keyboard to be used as the primarymethod for inputting text and data. While the implementation of the keyson a typewriter and, subsequently, computer keyboards have evolved frommechanical to electrical and finally to electronic, the size, placement,and mechanical nature of the keys themselves have remained largelyunchanged.

As computers evolved and graphical user interfaces developed, the mousepointer became a common user input device. With the introduction ofportable “laptop” computers, various new pointing devices were inventedas an alternative to the mouse, such as trackballs, joysticks, andtouchpads (also referred to “trackpads”). The overwhelming majority oflaptop computers now incorporate the touchpad as the primary pointingdevice (or touch-sensitive surfaces that are integrated with a display,such as a touch-sensitive display). As touch-sensitive displays haveincreased in popularity, numerous issues have emerged.

For example, combining the three primary user interface devices ofkeyboard, touchpad, and numpad into a single device results in thedevice becoming unreasonably large. The problem is further complicatedby the fact that many modern keyboards incorporate yet additional keysfor page navigation, multimedia controls, gaming, and keyboard settingsfunctions. The result can be a “keyboard” that is often larger than thecomputer itself.

Additionally, while the new paradigm of touch-centric computing has manyadvantages, one marked disadvantage is the lack of a keyboard. Althoughexternal physical keyboards can typically be connected to touch-screencomputers, it often defeats the purpose of the device and negates itsadvantages over traditional laptop computers.

As the evolution of computing devices has progressed toward touch-baseduser interfaces, a natural evolution for the idea of a keyboard has beento carry it into the virtual world of the computer display by designingonscreen keyboards. Smaller touchscreen devices such as PDAs andSmartphones don't have sufficient screen size to allow people to type onan onscreen keyboard using the conventional method of touch-typing withmultiple fingers. As a result, a plethora of inventions have sought toprovide alternative text input methods that require less physical spacethan a conventional keyboard layout. While these inventions have varyingbenefits for entering text on a small onscreen keyboard, they don'tallow text entry at speeds that compare to standard “ten-finger” typingon a conventional keyboard.

Thus, it is desirable to find a yet faster way for entering text thatmore closely matches the typing style learned on conventional keyboards.In doing so, there are three primary challenges: first, overcoming therelatively large amount of display real estate required for a 10-fingeronscreen keyboard. Second, overcoming the lack of tactile feedbackcommon in mechanical keyboards. And third, allowing the user to resttheir fingers on the “home-row” position on the onscreen keyboard, asthey normally would on a conventional electromechanical keyboard.

SUMMARY [0014] Disclosed herein are methods and systems that address theproblems identified above.

In one aspect, some embodiments disclosed herein help to disambiguateuser input by using finger encoding. In some embodiments, the locationof each finger of a user is identified and tracked as they type on anonscreen keyboard. The system learns which finger is typically used toselect each key on the keyboard for each user. Words are then encoded byidentifying which finger is used for each letter. This encoding can beused to disambiguate the word intended by the user, even if they typeinaccurately on the onscreen keyboard.

More specifically, in some embodiments, a method includes: monitoringtyping inputs received from a user at the touch-sensitive display;determining whether the typing inputs are converging towards a midpointof the touch-sensitive display or diverging away from the midpoint ofthe touch-sensitive display; in accordance with a determination that thetyping inputs are converging towards the midpoint of the touch-sensitivedisplay, providing a first feedback to the user to indicate that thetyping inputs are converging; and in accordance with a determinationthat the typing inputs are diverging away from the midpoint of thetouch-sensitive display, providing a second feedback to the user toindicate that the typing inputs are diverging. Additional detailsregarding such methods are provided below in reference to FIG. 16H.

In some embodiments, the display is a static display in which respectivekeys are painted on the touch-sensitive display. In these embodiments,reducing a display size for a respective key and/or changing displaypositions for respective keys is performed as a background process andthe displayed keys are not modified (in other words, respective keysremain at a same location on the touch-sensitive display, but regions ofthe display that are associated with selection of respective keys aremodified).

In another aspect, some embodiments disclosed herein help to ensure thatdisplay positions for keys on a keyboard are dynamically displayed atappropriate positions on a touch-sensitive display based on typingpatterns for a user. In some embodiments, a method includes: building arepository of words typed by a user on the touch-sensitive display,wherein each respective character of a respective word is associatedwith information identifying a finger that was used by the user toselect the respective character; presenting, on the touch-sensitivedisplay, a virtual keyboard with a plurality of keys; in response todetecting selection of a first key of the plurality of keys of thevirtual keyboard, determining a first finger that was used by the userto select the first key; after detecting selection of at least the firstkey and in response to detecting an ambiguous selection of an unknownkey of the plurality of keys of the virtual keyboard, determining asecond finger that was used by the user to select the unknown key;retrieving information associated with candidate words from therepository of words typed by the user, the information associated withthe candidate words including (i) a first character associated with thefirst key and information identifying the first finger as havingselected the first key and (ii) information identifying the secondfinger as having selected a different key that is distinct from thefirst key; and determining that the unknown key is a second key of theplurality of keys of the virtual keyboard based at least in part on theinformation associated with the candidate words. Additional detailsregarding such methods are provided below in reference to FIG. 16H.

In one more aspect, the physical space problem discussed above isaddressed by integrating the numeric keypad part of the keyboard and thetouchpad in the same physical location (e.g., a touch-sensitivesurface). This surface can be used to provide all the functions of thekeyboard, numpad, and touchpad, but in a much smaller space since itmakes it possible to “multiplex” or use the same physical space on thesurface for multiple functions. The touch surface may incorporate eithera dynamic or static display beneath it, or a mixture of both.

In some embodiments, the numeric keypad and the touchpad occupy the samephysical space. This is possible due to the fact that thetouch-sensitive surface, unlike traditional mechanical keys, can havethe spacing, size, orientation, and function of its “keys” dynamicallyassigned.

In some embodiments, the system has three modes of operation: numpadmode, touchpad mode, and auto-detect mode. A visual indicatorcommunicates with the user which mode it is in. The user changes themode via activation of a key or combination of keys on the keyboard.Visual indicators provide feedback to the user as to which mode thedevice is in.

In some embodiments, the system automatically determines which mode theuser intends based on their interaction with the touch surface. Forexample, if the user slides their finger across the surface, they mostlikely intend for it to act as a touchpad, causing the pointer to move.Similarly, if the user taps their finger on a specific sector of thetouch surface assigned to a number key, then they most likely intend forit to be used as a numpad.

In some embodiments, the system includes a surface with a multi-modearea, a plurality of touch sensors coupled to the surface, a pluralityof motion sensors, and a processor in signal communication with thesurface, the plurality of touch sensors, and the plurality of motionsensors is provided. The plurality of touch sensors are configured togenerate at least one sense signal based on sense user contact with thesurface. The plurality of motion sensors are configured to generate amotion signal based on sensed vibrations of the surface. The processoris configured to determine mode of operation associated with themulti-mode area based on interpretation of at least one of the generatedat least one sense signal and the motion signal associated with themulti-mode area.

In some embodiments, the modes of operation include at least two of akeyboard mode, a numeric keypad mode, or a touchpad mode.

In some embodiments, the processor is further configured to determinethe mode of operation based on a signal associated with a userselection.

In some embodiments, the surface includes a display device coupled tothe processor. In some embodiments, the user selection includesactivation of a mode key displayed by the processor on the surface.

In some embodiments, the surface includes at least one visual indicatorand the processor illuminates the at least one visual indicator based onthe determined mode of operation.

In some embodiments, the processor identifies a default mode ofoperation. In some embodiments, the processor identifies the defaultmode of operation to be the touchpad mode after an auto mode selectionhas occurred followed within a predefined amount of time by adetermination of a sliding motion at least on or near the multi-modearea based on the at least one sense signal. In some embodiments, theprocessor identifies the default mode to be the numeric keypad mode ifafter the auto mode selection no sliding motion is detected within thepredefined amount of time based on the at least one sense signal.

In some embodiments, the processor determines mode of operation to bethe touchpad mode, if the processor detects a touch-and-slide motion atthe multi-mode area based on the generated at least one sense signal andthe motion signal. In some embodiments, the processor determines mode ofoperation to be at least one of the numeric keypad mode or the keyboardmode, if the processor detects only a tap motion based on the generatedmotion signals and the detected tap motion did not occur within athreshold amount of time since the detected touch-and-slide motion.

In some embodiments, the processor returns interpretation of thegenerated at least one sense signal and the motion signal associatedwith the multi-mode area to the default mode after a predefined periodof time has expired since a previously generated at least one sensesignal and motion signal associated with the multi-mode area.

In some embodiments, the surface includes a display device coupled tothe processor and the processor is configured to generate an image andpresent the generated image in the multi-mode area of the surface,wherein the generated image is associated with current mode ofoperation.

In some embodiments, the surface includes a static representation of atleast one of a numeric keypad, keyboard or touchpad.

In another aspect, the embodiments disclosed herein also provide systemsand methods that allow the user to rest their fingers on the keys of anonscreen keyboard displayed on a touch-sensitive screen and dynamicallydefine the location, orientation, shape, and size of the onscreenkeyboard. Rather than the user having to take care to place theirfingers on the keys (which typically would require tactile markers onsaid keys), the system dynamically moves the location of the onscreenkeyboard to where the user's fingers are already resting.

In some embodiments, the process defines a “home-row definition event,”which is an action performed by the user that causes the system toredefine where the home-row of the onscreen keyboard is located. Thislocation is dynamically established based on the user's action.

In some embodiments, the home-row definition event is defined as theuser resting all four fingers of both hands simultaneously on thetouch-sensitive surface for a preset period of time (e.g., 1 second).

In some embodiments, the home-row definition event is defined as theuser double-tapping all four fingers of both hands on thetouch-sensitive surface and then resting them on the surface after asecond tap.

In some embodiments, the home-row definition event is defined as theuser resting all four fingers of both hands simultaneously on thetouch-sensitive surface and then pressing them down momentarily.

These actions (as well as others) are initiated by the user to indicateto the system that the user's fingers are in the home-row restingposition. The system then orients the onscreen keyboard accordingly.Note that the keys on the home-row needn't be in a continuous line (asthey are on most electromechanical keyboards). Rather, the location ofeach key on the home-row is defined by the placement of the user's eightfingers during a home-row definition event as sensed by touch sensors,and then extrapolated for keys that are not “home-row resting keys.” Inthis way the home-row could be along two separate lines, one for eachhand placement, or may even form two curves.

Once a home-row definition event has taken place, the system providesfeedback to the user in numerous ways. For example, the system providesvisual feedback by causing the onscreen keyboard to appear beneath theuser's fingers. As another example, the system provides an audible cue.In yet another aspect, the system causes the touch-screen to momentarilyvibrate.

In some embodiments, according to the user's preference, the onscreenkeyboard remains visible continuously while typing is taking place.Alternatively, the onscreen keyboard becomes transparent after thehome-row definition event. In another aspect, the onscreen keyboardbecomes semitransparent, allowing the user to see through the keyboardto content on the screen below.

In some embodiments, the onscreen keyboard cycles between visible andinvisible as the user types. Each time the user taps on the “hidden”onscreen keyboard, the onscreen keyboard temporarily appears and thenfades away after a user-settable amount of time.

In some embodiments, only certain keys become visible after eachkeystroke. The keys which become temporarily visible are those keys thatare most likely to follow the immediately preceding text input sequence(as determined by word and letter databases stored in the system).

In some embodiments, the onscreen keyboard becomes temporarily visiblewhen the user, with fingers resting in the home-row position, pressesdown on the surface with their resting fingers.

In some embodiments, the onscreen keyboard becomes visible when the userperforms a predefined action on the edge of the enclosure outside of thetouch sensor area, such as a double- or triple-tap.

In some embodiments, the home-row resting keys are defined as the eightkeys rested upon by the four fingers of each hand. In yet anotheraspect, the resting keys may be fewer than eight keys to accommodateusers who may not have use of all eight fingers.

In some embodiments, the system disambiguates which key was intendedaccording to movement of a particular finger in an intended direction.For example, the user lifts their ring finger and moves it slightlydownward and taps. The user may not have moved far enough to reach thevirtual location of the adjacent key, but their intention was clearly toselect it since they moved from their resting position by a definablethreshold distance and tapped in the direction of the adjacent key. Eventhough the tap may not have occurred on the adjacent key in thisexample, the system will select it.

In some embodiments, the system adjusts the probability of each keybeing selected, based on the text sequence that immediately preceded it.This probability is used in conjunction with the tap location algorithmdescribed in the previous paragraphs to determine the most likely keythe user intended to tap on.

In some embodiments, the system automatically accounts for “user drift”as they type on the onscreen keyboard. Without the benefit of tactilefeel for each key, it is easy for the user to move their hands slightlyas they type. The system tracks this behavior by comparing the center ofthe intended key with the actual location that the user tapped. If aconsistent drift is detected over the space of consecutive key events,the system shifts the location of the keys accordingly to accommodatethe drift. Again, rather than make the user take care where the keysare, the system moves the keys to where the user's fingers are alreadylocated.

If the user drifts too far to the point of straying off of thetouch-sensitive area, the system warns them with an audible, visual,and/or vibrating cue.

In another aspect, the method and system monitor for user taps that areon the surface of the portable computing device, but not within theboundaries of the touch sensor. For example, the user may tap an edge ofthe device's enclosure to indicate a spacebar actuation. As with othertap events, the system correlates the signals from the touch sensors andvibration sensors to determine the tap location. When an absence ofsignal is detected by the touch sensor, the system recognizes the eventas an “external tap” (i.e., a tap on the surface of the device, butoutside the boundaries of the touch sensors). External taps generateunique vibration waveforms depending on their location on the enclosure.Characteristics of these waveforms are stored in a database and are usedto uniquely identify the general location of the external tap. Theexternal taps, once identified, can be assigned to keyboard functions(such as space or backspace).

In some embodiments, a device including a display, a plurality of touchsensors coupled to the display, a plurality of motion sensors, and aprocessor in signal communication with the display, the plurality oftouch sensors, and the plurality of motion sensors is provided. In someembodiments, the plurality of touch sensors are configured to generatesense signals based on sensed user contact with the display. In someembodiments, the plurality of motion sensors are configured to generatemotion signals based on sensed vibrations of a housing. In someembodiments, the processor is configured to generate and present on thedisplay an image of a keyboard having a plurality of keys based on atleast one of the generated sense signals or the generated motionsignals. In some embodiments, the housing is configured to contain thedisplay, the plurality of touch sensors, the plurality of motionsensors, and the processor.

In some embodiments, the processor is configured to determine locationof the keyboard image on the display based on the generated sensesignals. In some embodiments, the processor is configured to determinelocation of the keyboard image on the display based on determination ofexistence of a home-row definition event. In some embodiments, theprocessor determines an existence of the home-row definition event whentwo or more generated sense signals are determined to be active for apredefined amount time.

In some embodiments, the processor is configured to: 1) determinelocations of home-row keys of the keyboard image based on determinationof location of the generated two or more sense signals; and 2) determinelocations of non-home-row keys of the keyboard image based on determinedlocation of at least one of the home-row keys.

In some embodiments, the processor is configured to: 1) determine sizesof home-row keys of the keyboard image based on determination oflocation of the generated two or more sense signals; and 2) determinesizes of non-home-row keys of the keyboard image based on determinedlocation of at least one of the home-row keys.

In some embodiments, the processor is configured to: 1) determineorientations of home-row keys of the keyboard image based ondetermination of location of the generated two or more sense signals;and 2) determine orientations of non-home-row keys of the keyboard imagebased on determined location of at least one of the home-row keys.

In some embodiments, the housing further includes a vibration deviceconfigured to generate vibrations at one or more frequencies. In suchembodiments, the processor is configured to cause the vibration deviceto activate at a predefined frequency based on the home-row definitionevent.

In some embodiments, the housing includes a vibration device configuredto generate vibrations at one or more frequencies. In such embodiments,the processor is configured to: 1) place the presented keyboard in astatic mode of operation; 2) determine location of at least one userfinger based on the sensor signal; and 3) cause the vibration device tocreate a vibration at a predefined frequency when the determinedlocation of the at least one user finger is within a threshold distancefrom the at least one home key.

In some embodiments, the vibration device is configured to alterintensity of the vibration based on distance of the at least one userfinger from the corresponding home key.

In some embodiments, the housing includes an audio device configured togenerate audio signals at one or more frequencies. In such embodiments,the processor is configured to: 1) place the presented keyboard in astatic mode of operation; 2) determine location of at least one userfinger based on the sensor signal; and 3) cause the audio device tocreate an audio signal at a predefined frequency when the determinedlocation of the at least one user finger is within a threshold distancefrom the at least one home key.

In some embodiments, the audio device is configured to alter intensityof the audio signal based on distance of the at least one user fingerfrom the corresponding home key.

In some embodiments, the processor is configured to: 1) periodicallyreceive sense signals associated with continual user finger contact withthe display; 2) determine if the received periodic sense signalsindicate drift from locations of the sense signals used during thegeneration and presentation of the keyboard image; and 3) move at leastone key of the keyboard image on the display based on a drift indicatedof the at least one key.

In some embodiments, the device includes an output device and theprocessor is configured to: 1) determine if the periodically receivedsense signals indicate user finger contact drift is within a thresholddistance of an edge of the display; and 2) output a signal to the outputdevice if user finger contact drift was determined to be within thethreshold distance.

In some embodiments, the processor is configured to: 1) sense a usertyping action based on the generated sense signals and the generatedmotion signals; and 2) change the keyboard image to be at least one ofsemitransparent or invisible when the user typing action is not sensedfor a predefined amount of time.

In some embodiments, after the keyboard image has been made at least oneof semitransparent or invisible, the processor is configured to causethe keyboard image to appear at least less transparent when a usertyping action has been sensed.

In some embodiments, the processor is configured to: 1) determine atleast one next most likely key to be activated based one or moreprevious key activations; and 2) uniquely display the determined atleast one next most likely key.

In some embodiments, the processor is configured to: 1) determinerelative movement of one or more user fingers from the home-row keysbased on the generated sense signals; and 2) generate a key activationevent based on the generated motion signals and the determined relativemovement.

In some embodiments, the processor is configured to: 1) generate one ormore candidate keys based on at least a portion of the generated sensesignals and the generated motion signals; and 2) generate a keyactivation event by disambiguating the generated one or more candidatekeys using a statistical probability model.

In some embodiments, the processor is configured to: 1) determine a sizevalue for at least one key based on statistical probability model and atleast one previous key activation event; and 2) alter the keyboard imagebased on the determined size value for the at least one key.

In some embodiments, the processor is configured to: 1) cause thepresented keyboard image to be invisible in an active state based on asensed first user action; and 2) cause the presented keyboard image tobe invisible in an inactive state based on a sensed second user action.

In some embodiments, the generated at least one motion signal isassociated with a location relative to the housing. In such embodiments,the processor is configured to identify a function based on the locationrelative to the housing, when the at least one motion signal has beengenerated and no sense signals have been generated.

In some embodiments, systems and methods that allow the user to resttheir fingers on a touch-sensitive surface and make selections on thatsurface by pressing are provided. Touch capacitance sensors thattypically provide X and Y location data associated with a user's touchare also used to discern finger pressure in the Z direction. This allowsthe user to make an actuation on the touch screen by simply pressingharder at a location where they may already be resting their finger(s).

In one aspect, the process discerns between the actions of tapping onthe surface, resting on the surface, and pressing on the surface. Itdoes so using, in part, thresholds for the touch signal (which may bedynamically altered to accommodate the touch signatures of differentusers). The process also takes into account the rate of the rising edgeof the touch signal to discern between a tap, a resting action, and apress.

It is desirable to allow a human user to rest their hands and/or fingerson a touch surface without causing an actuation, yet still allow otheractions issued by the user through touch, such as a press, to beinterpreted as commands by the system.

In some embodiments, a system including a touch-sensitive surface and aprocessor is provided. In some embodiments, the touch-sensitive surfaceincludes a plurality of touch capacitive sensors associated withactionable locations on the surface. In some embodiments, the processoris configured to: 1) determine a user interaction with the touchsensitive surface as a resting action based on one or more signalsreceived from one or more of the plurality of touch sensors, wherein thesignals are above a first threshold value; and 2) determine a userinteraction with the touch sensitive surface as a press action based onthe one or more signals received from one or more of the plurality oftouch sensors, wherein the received signals are above a second thresholdvalue.

In some embodiments, the first and second threshold values arepredefined. In other embodiments, the first and second threshold valuesare variable based on individual touch characteristics of each user.

In some embodiments, the processor is configured to assert an activationafter a determined resting action for a particular user interaction isfollowed by a determined press action on an actionable location.

In some embodiments, the processor is further configured to determine apress and hold event when a determined resting action for a particularuser interaction is followed by a determined press action that issustained for longer than a predefined key repeat time.

In some embodiments, the processor is further configured to: 1)determine a user interaction as a selection event based on one of thesignals having a leading rising edge with a rate-of-change that exceedsa first rate-of-change threshold followed within a predefined amount oftime by the signal decreasing in value at a rate-of-change greater thana second rate-of-change threshold. In some embodiments, the first andsecond rate thresholds are the same.

In some embodiments, the amplitude of the signal is greater than atleast the first threshold value.

In some embodiments, the processor is further configured to: determinethe initial user interaction with the touch sensitive surface is theresting action is further determined when a rate-of-change of theleading rising edge of the signal is less than the first rate-of-changethreshold and the one or more signals are above the first thresholdvalue.

In some embodiments, the processor is further configured to: determinethe initial user interaction with the touch sensitive surface is thepress action is further determined when the rate-of-change of theleading rising edge of the signal is less than the first rate-of-changethreshold and the one or more signals are above the second thresholdvalue.

In some embodiments, the system includes an output device configured topresent a response corresponding to the press action.

In another aspect, a system including a touch-sensitive surface and aprocessor in signal communication with the touch-sensitive surface isprovided. In some embodiments, the touch-sensitive surface includes aplurality of touch capacitive sensors associated with actionablelocations on the surface and the sensors are configured to generate oneor more signals. In some embodiments, the processor is configured to: 1)determine a user interaction with the touch sensitive surface is aresting action based on one or more signals received from one or more ofthe plurality of touch sensors, wherein the signals are above a firstthreshold value; 2) after determination of the resting action, determinea peak of the one or more signals and determine a difference inamplitude of a location of the one or more signals associated with theresting action and the determined peak; 3) if the determined differenceis greater than a first predefined delta threshold, determine that auser interaction with the touch sensitive surface is a press action; 4)after determination of the press action, determine that a userinteraction with the touch sensitive surface is at least one of: (a) arest and press release action, if the one or more signals are determinedto be at or below the first threshold value; or (b) a press releaseaction, wherein the processor determines an amplitude of a trough of theone or more signals and determines the user interaction is the pressrelease action if the trough amplitude and the determined peak have adifference that is greater than a second predefined delta threshold.

In some embodiments, the first and second predefined delta thresholdsare the same.

In some embodiments, the system further includes an output deviceconfigured to present a response corresponding to the press action.

In yet another aspect, a system including a touch-sensitive surface anda processor is provided. In some embodiments, the touch-sensitivesurface includes a plurality of touch capacitive sensors associated withactionable locations on the surface, the sensors configured to generateone or more signals. In some embodiments, the processor is configuredto: 1) determine the initial user interaction with the touch sensitivesurface is the resting action is further determined when arate-of-change of the leading rising edge of the signal is less than thefirst rate-of-change threshold and the one or more signals are above thefirst threshold value; 2) determine the initial user interaction withthe touch sensitive surface is the press action is further determinedwhen the rate-of-change of the leading rising edge of the signal is lessthan the first rate-of-change threshold and the one or more signals areabove the second threshold value; 3) after determination of the restingaction, determine a peak of the one or more signals and determine adifference in amplitude of a location of the one or more signalsassociated with the resting action and the determined peak; 4) if thedetermined difference is greater than a first predefined deltathreshold, determine that a user interaction with the touch sensitivesurface is a press action; 5) after determination of the press action,determine that a user interaction with the touch sensitive surface is atleast one of: (a) a rest and press release action, if the one or moresignals are determined to be at or below the first threshold value; (b)or a press release action, wherein the processor determines an amplitudeof a trough of the one or more signals and determines the userinteraction is the press release action if the trough amplitude and thedetermined peak have a difference that is greater than a secondpredefined delta threshold.

In some embodiments, the first and second predefined delta thresholdsare the same.

In some embodiments, the system further includes an output deviceconfigured to present a response corresponding to the press action.

In an additional aspect, some embodiments disclosed herein detect inputsbefore contact is made with a touch-sensitive display, in order toprovide for improved typing accuracy and disambiguation of ambiguouskeystrokes. For example, in some embodiments disclosed hereintouchscreen typing is improved by dynamically positioning displayed keysbelow a user's fingers while the user's fingers are hovering above thetouchscreen surface, thus improving typing accuracy and rendering itunnecessary for the user to look at the keyboard before starting totype. In other embodiments, touchscreen typing is further improved bygenerating a database of finger-to-key associations, and using thatdatabase to resolve an otherwise ambiguous user touch to an intendedkeystroke. The database of finger-to-key associations may be generated,for example, by identifying positions of a user's hovering fingers priorand applying the finger identifications in recording which of the user'sfingers is used to stroke respective keys of a displayed keyboard. Forexample, as discussed below in reference to method 1680 and FIG. 16Hbelow.

The locations of each of the user's fingers while hovering above thesurface are determined by using a hover detection system that may alsobe combined with user-specific hand print information. This hand printmay be used, alone or together with other information, a user's home-rowposition and is established by using touch sensors to measure andcapture the finger locations relative to each other when a user reststheir hands on the touch surface. Correlation between the hover sensordata and touch sensor data increases the confidence level by which thesystem can tell which of the user's fingers were used to select a key.By knowing which finger was used to make the selection, the system canmore accurately disambiguate which letter the user intended to type.

First, the system determines the location of the user's fingers, whichmay be hovering above the touch surface, resting on the touch surface,or a combination of both resting and hovering.

Next, the system may automatically position the home-row keys of theonscreen keyboard directly below the hovering or resting fingers bycomparing the location of the fingers with the predetermined user's homerow position, defined as the user's “hand print”. (This feature may beset as a user option).

Next, the system determines the typing style of the user: two-fingertypist, full touch-typist (8, 9, or 10 fingers), or hybrid typist (3, 4,5, 6, or 7 fingers). The system then maps keys of the keyboard to thespecific fingers of the user that are used to type those keys. Thedefault finger allocation may originate from a default allocation storedby the system, but the system may also learn from the typing style ofthe user and adapt the finger allocation table dynamically. The fingerallocation table for each specific user may be stored in a database andused by the system to form a set of typing characteristics called a“Type Print” used to identify a specific user (analogous to afingerprint). See, for example, U.S. patent application Ser. No.13/485,802, filed May 31, 2012 (entitled “System for Detecting a User ona Sensor-Based Surface”) which is hereby incorporated by reference inits entirety. This user-specific data may be stored locally on thedevice or in the “cloud.” The keys of the keyboard may or may not bevisibly displayed to the user. In a preferred embodiment, the keysremain virtual and invisible on the display so-as not to obscure othergraphical information depicted on the display.

Next, the system continuously tracks the location of each of the user'sfingers and correlates which finger was used to perform key actuationson the onscreen keyboard. The continuous tracking of each fingerlocation takes place when the fingers are hovering above the touchsurface, resting on the touch surface, or a combination of both.

When a keystroke is made in an unambiguous way and the user is deemed tohave accepted the input by not backspacing or correcting it, then thesystem records which finger was used to make that selection. Thisinformation is stored in a database and may be used later to helpdisambiguate other keystrokes.

When an ambiguous key selection takes place, the system determines whichfinger was used to make the selection, references the pre-storeddatabase to determine what keys were previously typed with that finger,and then uses that information to disambiguate the key selection.

In some embodiments, the process discussed above may also work inreverse. If the system is unable to make an unambiguous determination asto which finger was used to make a key selection, it refers to previousselections of that same key and infers from that historical record whichfinger was most likely used to make the current selection.

In some embodiments, typing on touchscreens is improved by dynamicallyand automatically positioning the desired home-row keys of an onscreenkeyboard below the user's fingers while their fingers are hovering abovethe surface, thus reducing the need for the user to look at the onscreenkeyboard while typing. The location of each of the user's fingers whilehovering and/or resting is monitored. Correlation between the hoversensor data and touch sensor data increases the confidence level bywhich the system can determine which of the user's fingers were used toselect a key. By determining which finger was used to make theselection, the system accurately disambiguates which letter the userintended to type. Furthermore, hover sensor data can greatly help tosolve the problem of false positive key actuations (i.e., unintended keyactuations by a user during a set-down event). The hover sensors canreport that all fingers are approaching the surface more or lesssimultaneously and can advise the system that a set-down event isimminent. This eliminates the problem of false-positives during set-downevents.

Methods to improve keyboard display locations based on hover data arealso disclosed herein. For example, a method of operation within acomputing device having a touchscreen is disclosed. The method includes:rendering a keyboard on the touchscreen, including rendering individualkeys of the keyboard in respective positions on the touchscreen; sensingat least one hand of a user hovering at a location above a surface ofthe touchscreen; and adjusting the positions of the individual keys onthe touchscreen surface based on the location of the at least one handof the user.

In some embodiments, sensing at least one hand of the user hovering at alocation above the surface of the touchscreen includes sensing one ormore fingers of the user hovering above the surface of the touchscreen.

In some embodiments, sensing at least one hand of the user includessensing the at least one hand of the user via a proximity sensor of thecomputing device.

In some embodiments, rendering individual keys of the keyboard inrespective positions on the touch screen includes displaying, as each ofthe individual keys, a graphical image of the individual key, thegraphical image having a shape, area, locus and angular orientation, andadjusting the display positions of the individual keys includes,changing at least one of the shape, area, locus or angular orientationof the individual key.

In some embodiments, rendering individual keys of the keyboard inrespective positions on the touch screen includes rendering, as each ofthe individual keys, a virtual depiction of the individual key, withouta corresponding displayed graphical image, but having a shape, area,locus and angular orientation, and adjusting the positions of theindividual keys includes, changing at least one of the shape, area,locus or angular orientation of the individual key.

In some embodiments, sensing at least one hand of a user hovering at thelocation above the surface of the touchscreen includes sensing bothhands of the user hovering at respective first and second locationsabove the surface of the touchscreen, wherein the first and secondlocations are offset from one another along an axis parallel to thesurface of the touchscreen. In these embodiments, adjusting thepositions of the individual keys on the touchscreen surface includesoffsetting the display positions of a first set of the individual keysassociated with one of the user's hands from positions of a second setof the individual keys associated with the other of the user's hands inaccordance with the offset between the first and second locations of theuser's hands.

In some embodiments, the method further includes: generating a handprintof the at least one hand of the user touching the touchscreen. In theseembodiments, sensing the at least one hand of the user hovering at thelocation above the surface of the touchscreen includes determininglocations of one or more fingers of the user's hand based at least inpart on the handprint.

In some embodiments, adjusting the positions of the individual keys onthe touchscreen surface based on the location of the at least one handof the user includes adjusting the positions of the individual keysbased at least in part on the locations of the one or more fingers ofthe user's hand.

In some embodiments, rendering the keyboard on the touchscreen includesat least one of (i) displaying the keyboard on the touchscreen or (ii)enabling a region of the touchscreen to function as a keyboard withoutdisplaying the keyboard, and wherein rendering individual keys of thekeyboard in respective positions on the touchscreen includes at leastone of displaying the individual keys of the keyboard in respectivepositions on the touchscreen or (ii) enabling respective regions of thetouchscreen to function as the individual keys of the keyboard withoutdisplaying the individual keys.

Another method is provided that operates within a computing devicehaving a touchscreen. The method includes: tracking locations of auser's fingers above the touchscreen while the user types on a keyboardon the touchscreen; and generating, based on the locations of the user'sfingers above the touchscreen while the user types and characters typedby the user, finger-to-key correlation information that indicates, foreach of a plurality of individual keys of the keyboard on thetouchscreen, which of the user's fingers is most likely to be used totype that key.

In some embodiments, the method further includes: disambiguating a usertouch on the touchscreen as a keystroke directed to one of a pluralityof possible individual keys of the keyboard displayed on the touchscreenbased, at least in part, on the finger-to-key correlation information.

In some embodiments, tracking locations of the user's fingers above thetouchscreen includes tracking locations of the user's fingers while theuser's hands are hovering above a surface of the touchscreen.

In some embodiments, tracking locations of the user's fingers while theuser's hands are hovering above a surface of the touchscreen includessensing locations of the user's hands via a proximity sensor of thecomputing device.

In some embodiments, the method further includes: generating a handprintof at least one hand of the user, and, in these embodiments, trackinglocations of the user's fingers while the user's hands are hoveringabove a surface of the touchscreen includes tracking locations of theuser's fingers based at least in part on the handprint.

In some embodiments, the method further includes: disambiguating whichof the user's fingers was used to touch a key in a non-ambiguouskeystroke based at least in part on the finger-to-key correlationinformation.

BRIEF DESCRIPTION OF THE DRAWINGS

The implementations disclosed herein are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings. Like reference numerals refer to corresponding partsthroughout the drawings.

FIG. 1 is a block diagram of an exemplary system formed in accordancewith some embodiments.

FIG. 2 is a graphical representation of a state machine, detailing thestates of resting and pressing, in accordance with some embodiments.

FIG. 3 is a data flow diagram of exemplary processes performed by thesystem shown in FIG. 1, in accordance with some embodiments.

FIGS. 4A and 4B are plots of waveforms representing the touch signalvalue in the time domain for various press actions, in accordance withsome embodiments.

FIG. 5 illustrates the disruption of an electrical field caused by thecapacitance of a lightly-touching finger, in accordance with someembodiments.

FIG. 6 illustrates the disruption of an electrical field caused by thecapacitance of a finger being pressed strongly into the surface, inaccordance with some embodiments.

FIGS. 7A, 7B, and 7C are waveform plots of a tap selection, a rest, anda press action, all in the time domain, in accordance with someembodiments.

FIG. 8 shows an exemplary process performed by the system shown in FIG.1, in accordance with some embodiments.

FIG. 9A is a schematic of a partial view of an exemplary touch-sensitivesurface formed, in accordance with some embodiments.

FIG. 9B is a schematic of a touch-sensitive surface used to illustrateplacement of a user's fingers on a keyboard, in accordance with someembodiments.

FIGS. 10A through 10F show a flowchart of exemplary processes performedby the system shown in FIG. 1, in accordance with some embodiments.

FIG. 11A is a schematic view of a tablet device with a flat-surfacedvirtual keyboard, in accordance with some embodiments.

FIGS. 11B and 11C are schematics of a touch-sensitive surface withonscreen keyboard displays, in accordance with some embodiments.

FIG. 12 is an overview of the sequential actions taken by the systemaccording to one embodiment during a typical user interaction of typing,in accordance with some embodiments.

FIG. 13A is a schematic view of a portable computer with a touch-screendisplay on which a keyboard system is displayed, in accordance with someembodiments.

FIG. 13B shows touch points created by a user resting their fingers andhands on a touch-sensitive surface, in accordance with some embodiments.

FIG. 14A is a visual representation of typical touch points created by auser resting all fingers and thumbs of the left and right handsuniformly on a touch screen, in accordance with some embodiments.

FIG. 14B is a visual representation of touch points created by a userresting their hands and fingers in a non-uniform manner on a touchscreen, in accordance with some embodiments.

FIG. 14C shows one example mapping of fingers to finger identifiers, inorder to help build finger-to-key encodings in accordance with someembodiments.

FIGS. 15A through 15F shows the typical positioning of human fingers forthe left and right hands as they are resting on a touch surface invarying finger combinations, in accordance with some embodiments.

FIGS. 16A through 16G show an embodiment of a software algorithm toidentify and track the fingers of a user resting or hovering above atouch surface, in accordance with some embodiments.

FIG. 16H is a flowchart depicting a method 1680 of disambiguating keyactuations using finger-to-keys encoding information, in accordance withsome embodiments

FIGS. 17A and 17B is a visual representation of the touch signals thatmay be sensed as a user hovers the fingers of their right hand above atouch-sensitive surface, in accordance with some embodiments.

FIGS. 18A and 18B is a simplified view of the touch signals that may besensed as a user hovers the fingers of their right hand above atouch-sensitive surface, in accordance with some embodiments.

FIG. 19 is a sample plot of data collected from numerous human subjectsshowing the distance ratio between each finger pair, in accordance withsome embodiments.

FIG. 20 is a depiction of an onscreen keyboard on which the word “in_”is ambiguously typed, in accordance with some embodiments.

FIG. 21 shows a preferred embodiment of a software algorithm to storeinformation indicating which finger is used to select keys in a databaseand then use that database to help disambiguate inaccurate typing, inaccordance with some embodiments.

FIG. 22 is a flowchart depicting a method 2200 of adaptively presentinga virtual keyboard, in accordance with some embodiments.

FIGS. 23A-23E and FIGS. 24A-24D provide user interface examples thatillustrate some aspects of the method 2200.

DETAILED DESCRIPTION

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings. In the following description,numerous specific details are set forth in order to provide anunderstanding of the various described embodiments. However, it will beapparent to one of ordinary skill in the art that the various describedembodiments may be practiced without these specific details. In otherinstances, well-known methods, procedures, components, circuits, andnetworks have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are used onlyto distinguish one element from another. For example, a first devicecould be termed a second device, and, similarly, a second device couldbe termed a first device, without departing from the scope of thevarious described embodiments. The first device and the second deviceare both devices, but they are not the same device.

The terminology used in the description of the various embodimentsdescribed herein is for the purpose of describing particular embodimentsonly and is not intended to be limiting. As used in the description ofthe various described embodiments and the appended claims, the singularforms “a,” “an,” and “the” are intended to include the plural forms aswell, unless the context clearly indicates otherwise. It will also beunderstood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“includes,” “including,” “comprises,” and/or “comprising,” when used inthis specification, specify the presence of stated features, integers,steps, operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context. Asused herein, the term “exemplary” is used in the sense of “serving as anexample, instance, or illustration” and not in the sense of“representing the best of its kind.”

FIG. 1 shows a block diagram of an exemplary device 100 (comprisinghuman interface hardware components, memory, and processors) forproviding a touch interface that, in some embodiments, discerns betweentapping, resting, and pressing, provides a multiplexed numeric keypadand touchpad, and/or provides an adaptive onscreen keyboard userinterface for alphanumeric input. The device 100 includes one or moretouch sensors 120 that provide input to a CPU (processor) 110. The touchsensors 120 notify the processor 110 of contact events when a surface istouched. In one embodiment, the touch sensor(s) 120 (e.g., capacitivetouch sensors or a pressure sensor, such as a strain gauge), or theprocessor 110, include a hardware controller that interprets raw signalsproduced by the touch sensor(s) 120 and communicates the information tothe processor 110, using a known communication protocol via an availabledata port. In some embodiments, the processor 110 generates an imagethat is presented on an optional display 130 (touch surface) oralternatively, the display may be static. In some embodiments, theprocessor 110 communicates with an optional hardware controller to causethe display 130 to present an appropriate image.

In some embodiments, the touch sensors 120 include capacitive touchsensors. In some embodiments, the analog values produced by thecapacitive touch sensors are obtained and analyzed (e.g., by processor110 of device 100). When a user presses their finger against the display130 (e.g., a display with a touch-sensitive surface or a touchscreendisplay), more of their finger comes in contact with the surface as theflesh of their fingers flattens against the surface. In someembodiments, this increase in contact causes a corresponding change inthe touch capacitive signal (e.g., the analog value produced by thecapacitive touch sensors of touch sensors 120). In this way, a press isdetected on the touch-sensitive surface.

In some embodiments, the touch sensors 120 include pressure sensors(e.g., a strain gauge) within device 100 or a component thereof (such asan external keyboard). The sensed weight (or pressure) of a contact on atouch-sensitive surface increases as the user presses on a key (e.g., asoft keyboard displayed on display 130). In some implementations, tolocalize the press, more than one strain gauge is used and the resultingsignals detected by each strain gauge are used to determine a locationon the touch-sensitive surface corresponding to the press. In someembodiments, the determined location is correlated with additional touchsensor data (e.g., data obtained by capacitive touch sensors included intouch sensors 130) to further refine the location of the press. In someembodiments, the device 100 includes pressure sensors and capacitivetouch sensors and device 100 detects occurrence of a press based onoutput from the pressure sensors and then determines a location on thetouch-sensitive surface corresponding to the press using the change incapacitance of the capacitive touch sensors.

In some embodiments, touch sensors 120 include resistive sensors thatare used to determine/detect press actions on the touch-sensitivesurface. More specifically, as the user touches the touch-sensitivesurface, the impedance between two planes on the surface changes andthis change is used to detect the different between a rest and a press.

In some embodiments, the device 100 optionally includes one or morevibration sensors 125 (e.g., accelerometers) that communicate signals tothe processor 110 when the surface is tapped, in a manner similar tothat of the touch sensor(s) 120. The processor 110 generates a keyboardimage that is presented on the display 130 (e.g., a display with atouch-sensitive surface) based on the signals received from the sensors(e.g., touch sensors 120 and vibration sensors 125). A speaker 135 isalso optionally coupled to the processor 110 so that any appropriateauditory signals are passed on to the user as guidance (e.g., errorsignals). A vibrator 180 is also optionally coupled to the processor 110to provide appropriate haptic feedback to the user (e.g., errorsignals).

In some embodiments, the processor 110 is in data communication with amemory 140, which includes a combination of temporary and/or permanentstorage, and both read-only and writable memory (random access memory orRAM), read-only memory (ROM), writable nonvolatile memory, such as FLASHmemory, hard drives, floppy disks, and so forth. The memory 140 includesprogram memory 150 that includes all programs and software such as anoperating system 151, press detection software component 152, adaptiveonscreen keyboard (“OSK”) software component 154, User GestureRecognition software 155, and any other application software programs153. The memory 140 also includes data memory 160 that includes SystemSettings 161, a record of user options and preferences 162 (e.g.,required by the User Gesture Recognition software 155), statisticaldatabase(s) 181 (e.g., word database(s)), and any other data 163required by any element of the device 100.

In some embodiments, the device 100 allows the user to perform at leastthree interactions on the touch-sensitive surface of display 130 (alsoreferred to as a touchscreen or a touch surface): a touch-and-releaseselection (or a “tap”), a resting action wherein they rest two or morefingers simultaneously on the touch surface, and a pressing action.Being able to distinguish between these three actions significantlyimproves the flexibility and usefulness of the user interface of thedevice 100. For example, the touch surface can be used as a keyboard,allowing the user to rest their fingers on it as they would whiletouch-typing on a traditional keyboard.

In some embodiments, once a home-row event has been detected by theprocessor 110 based on signals from the sensors (e.g., touch sensors 120and vibration sensors 125), the processor 110 positions a virtualonscreen keyboard beneath the user's fingers on the display 130. As theuser types, the processor 110 constantly monitors the placement of theuser's fingers, as well as tapped locations for each key actuation, andmakes adjustments to the location, orientation, and size of each key(and the overall keyboard) to ensure the onscreen keyboard is locatedwhere the user is typing. In this way, it is possible to account for theuser's “drifting”, or moving their fingers off of the original positionof the onscreen keyboard. If the user drifts too far in one directionso-as to reach the edge of the touch sensor area, the processor 110outputs an audible and/or haptic warning.

At any time, the user may manually re-assign the location of theonscreen keyboard by initiating a home-row definition event (asdescribed above).

In one embodiment, haptic feedback is provided via the vibrator 180 whenthe user positions their index fingers on the keys commonly-referred toas the “home keys” (F and J keys on a typical English keyboard). In oneembodiment, a momentary vibration is issued when the user rests theirfingers on the keys using a slightly different frequency of vibrationfor left and for right. In this manner, the user may choose to movetheir hands back into a fixed home-row position, when the user choosesto manually re-assign the location of the onscreen keyboard byinitiating a home-row definition event (in other words, processor 110does not dynamically change the position of the onscreen keyboard, thelocation of the onscreen keyboard instead changes in response to theuser's initiation of a home-row definition event). In anotherembodiment, the intensity of these vibrations may change depending uponfinger position relative to the home keys of the fixed home-row.

The device 100 allows the user to type without looking at their fingersor the virtual keyboard. It follows, then, that the keyboard need not bevisible at all times. This allows valuable screen space to be used forother purposes.

In one embodiment, the visual appearance of the keyboard varies itsstate between one or more of the following: visible, partially visible,invisible, and semitransparent. The full keyboard visually appears whena home-row definition event takes place or when the user has restedtheir fingers without typing for a settable threshold amount of time. Asthe user begins to type, the keyboard fades away to invisible until theuser performs any one of a number of actions including, but not limitedto: a home-row definition event, pausing typing, pressing on fourfingers simultaneously, or some other uniquely identifying gesture. Inanother embodiment, the keyboard does not fade away to be completelyinvisible, but rather becomes semitransparent so the user can stilldiscern where the keys are, but can also see content on the display thatis “beneath” the onscreen keyboard.

In one embodiment, the keyboard temporarily “lights”, or makes visible,the tapped key as well as those that immediately surround the tapped keyin a semitransparent manner that is proportional to the distance fromthe tapped key. This illuminates the tapped region of the keyboard for ashort period of time.

In one embodiment, the keyboard becomes “partially” visible with thekeys having the highest probability of being selected next lighting upin proportion to that probability. As soon as the user taps on a key,other keys that are likely to follow become visible or semi-visible.Keys that are more likely to be selected are more visible, and viceversa. In this way, the keyboard “lights” the way for the user to themost likely next key(s).

In one embodiment, the onscreen keyboard is made temporarily visible bythe user performing tap gestures (such as a double- or triple-tap inquick succession) on the outer rim of the enclosure surrounding thetouch-sensitive surface.

The various modes of visual representation of the onscreen keyboard maybe selected by the user via a preference setting in a user interfaceprogram (e.g., by modifying a preference setting stored in useroptions/preferences 162).

FIG. 2 is a state diagram that illustrates how a press state isdetermined by the processor 110. The system is initialized in 200 andthen enters the idle state 205 where no touch is detected. When a touchsignal is detected, the system begins to measure the accumulation of thesignal. When the accumulation reaches a pre-defined threshold called theBinary Rest Threshold in 206, the system proceeds to the Plateau State210. In the Plateau State 210, the user is deemed to be resting theirfinger(s) on the touch surface. If the user removes their finger(s) fromthe surface and the Slope Accumulation drops below the Binary RestThreshold in 211 then the system returns to Idle State 205. From thePlateau State 210 a user may press their finger harder into the surfacecausing the Slope Accumulation to continue to increase past apre-defined Positive Press Threshold 212, upon which the system proceedsto the Positive Press Detect State 215 and asserts a press action. Aslong as the user maintains the pressure while in the Positive PressDetect State 215, the system maintains the press assertion (similar toholding down a key on a traditional keyboard). Once in the PositivePress Detect State 215, the user may lift their finger(s) from thesurface causing the Slope Accumulation to decrease below the Binary RestThreshold in 217 and the system returns once again to the Idle State205. However, while in the Positive Press Detect State 215, the user mayreduce the pressure of the pressing action without completely removingtheir finger. In this case, a negative inflection point occurs where thetouch signal decreases to a point and then either levels out or beginsto increase again (i.e. where the slope of the touch signal curve iszero as it passes from negative to positive). When a negative inflectionpoint is detected the system determines if the Slope Accumulation hasdecreased below a Negative Press Threshold point in 216, at which pointthe system advances to the Negative Press Detect State 220 and the pressaction is released. Note that the Negative Press Detect State 220 issimilar to the Plateau State 210 in that the user is deemed to beresting. However, the absolute value of the touch signal may be quitedifferent between the two states. When in the Negative Press DetectState 220 the system watches for a maximum inflection point (where theslope of the curve is zero as it passes from positive to negative). Whena max inflection point takes place and the Slope Accumulation exceedsthe Positive Press Threshold in 221, the system returns to the PositivePress Detect State 215 and asserts a press action. Alternatively, whilein the Negative Press Detect State 220, if the Slope signal falls belowthe Binary Rest Threshold in 222 then the user is deemed to have liftedtheir finger off the surface and the system returns to the Idle State205.

FIG. 3 is a data flow diagram that shows how the CPU 110 measures,stores, and analyzes the touch signal. In block 300 the system acquiresthe raw sensor data from an analog to digital convertor (ADC). Thesignal is then passed through a low-pass filter in block 305 in order tosmooth out any high frequency noise that may be present in the signal.The result is then stored in a Cache (2) in block 310. The slope of thesignal is then analyzed in block 315, followed by detection of theminimum and maximum inflection points of the signal in block 320. Inblock 325 the system accumulates the slope changes and stores the resultin Cache (1) in block 330. This calculation determines the amplitudedifference between the min and max inflection points. In block 335, therate of change of the signal is determined and stored in Cache (1) inblock 340. The rate of change of the signal is helpful in determiningthe difference between a tap selection, a resting set-down action, and apress (as illustrated in FIGS. 7A, 7B, and 7C). In block 345 of FIG. 3,the system determines the current press state.

FIGS. 4A and 4B are representations of the touch signal going through anumber of conditions resulting in press actions being issued by thesystem. In FIG. 4A the system follows a very simple process of usingfixed threshold values to determine the difference between a restingaction and a press. The user touches the surface at 4000 causing thetouch signal to rise above the pre-defined Rest Threshold 4050, at whichpoint the signal levels off at 4010 causing an inflection point andputting the system into the Plateau State 210. Sometime later, the userpresses harder on the surface causing the touch signal to increase abovethe Press Threshold 4055 to a local maxima value at 4020 at which pointthe system asserts a press action (indicated by the black circle). Thesystem continues looking for maxima and minima inflection points. Theinflection points found at 4025 and 4030 are ignored since they occurabove the Press Threshold, meaning the press asserted at 4020 continuesto be asserted. At 4035 the system detects a minima inflection pointthat falls above the Rest Threshold 4050 and below the Press Threshold4055 at which point it asserts a press release action (indicated by thehollow circle). The user then presses again causing the touch signal toincrease past the Press Threshold. The system detects the maximainflection point at 4040 and assets another press action. The user thencompletely lets go, causing the touch signal to fall back to zero.Although no inflection point is detected, at 4045 the system recognizesthat the touch signal has fallen below the Rest Threshold 4050 andassets a press release action.

The method described in the above paragraph associated with respect toFIG. 4A is straight-forward, but fails to discern the possible pressaction that takes place between 4025 and 4030. When a user performsmultiple presses in quick succession, the touch signal often remainsabove the Press Threshold even on the press release action. In order toremedy this short-coming an embodiment is illustrated in FIG. 4B.

Referring to FIG. 4B, the user touches the surface at 4100 causing thetouch signal to rise above a pre-defined Rest Threshold 4150, at whichpoint the signal levels off at 4110 causing an inflection point whichthe system discerns as a Rest assertion and places the state machineinto the Plateau State 210. Sometime later, the user presses harder onthe surface causing the touch signal to increase to a local maximumvalue at 4120. The relative change in the signal from 4110 to 4120 iscompared with another threshold called the Press Assertion DeltaThreshold. If the increase in signal between 4110 and 4120 is greaterthan the Press Assertion Delta Threshold then a press action is assertedby the system at 4120 (indicated by the solid black circle). Followingthis assertion, the user decreases the touch pressure between 4120 and4125 but then once again increases the pressure between 4125 and 4130.At 4125, the system detects a minimum inflection point and measures thechange in the touch signal between 4120 and 4125 which is then comparedwith yet another threshold called the Press Release Delta Threshold. Ifthe absolute value of the decrease in the touch signal between 4120 and4125 is greater than the Press Release Delta Threshold then a releaseaction is asserted by the system (indicated by the hollow circle). Asimilar process takes place between 4130, 4135, and 4140 only withdifferent amplitudes and rate of change in the signal. Finally, the userstops pressing at 4140 but keeps their finger in contact with thesurface in a resting action at 4145, at which point the system asserts apress release action. After some amount of time, the user then removestheir finger from the touch surface and the signal quickly falls tozero. As the signal decreases through the Rest Threshold the systemasserts a Rest release action at 4150.

In one embodiment the two methods described in FIG. 4A and FIG. 4B maybe selectively combined.

FIG. 5 illustrates one of many possible embodiments in how atouch-sensitive surface can be implemented using capacitance. Atouch-sensitive surface 500 is made up of one or more sensors in whichan electrode 510 emits an electrical signal forming an electrical field530, 540, and 570. An adjacent electrode 520 couples with a portion ofthe formed electrical field 570. The coupled signal at the adjacentelectrode 520 is detected and measured by the system. As a human finger550 touches the surface 500, a portion of the electrical field 540couples with the finger, resulting in less of the electrical field 570coupling with the second electrode 520. The processor 110 receives adigital representation of the analog voltage measurement obtained fromthe second electrode 520 then detects the change of the signal at thesecond electrode 520 and determines a touch has taken place. The degreeto which the electrical field 540 couples with the human finger 550 isdependent, in part, on the amount of surface area 560 with which thefinger comes in contact. A “light” touch is shown in FIG. 5 where thefinger 550 is just making contact with the touch surface 500. Arelatively lower amount of the electrical field 540 is disrupted by thelight touch.

FIG. 6 illustrates the effects of a stronger press on the touchcapacitance signals. A touch-sensitive surface 600 is made up of one ormore sensors in which an electrode 610 emits an electrical signalforming an electrical field 630, 640, and 670. An adjacent electrode 620couples with a portion of the formed electrical field 670. The coupledsignal at the adjacent electrode 620 is detected and measured by thesystem. As a human finger 650 presses hard on the surface 600, arelatively larger portion of the electrical field 640 couples with thefinger, resulting in less of the electrical field 670 coupling with thesecond electrode 620. The processor 110 receives a digitalrepresentation of the analog voltage measurement obtained from thesecond electrode 620 then detects the change of the signal at the secondelectrode 620 and determines a press has taken place. The degree towhich the electrical field 640 couples with the human finger 650 isdependent, in part, on the amount of surface area 660 with which thefinger comes in contact. A “heavy” touch, or press, is shown in FIG. 6where the finger 650 makes strong contact with the touch surface 600causing the finger to flatten out at 660. A relatively larger amount ofthe electrical field 640 is disrupted by the pressing action.

FIGS. 7A, 7B, and 7C illustrate the three actions of a tap selection, aresting set-down action, and a set-down press action respectively. Boththe amplitude of the touch signal and the slope of the leading edge ofthe signal are used to determine which action is being initiated by theuser. In FIG. 7A the user quickly taps on a key causing the signal toexceed a pre-defined first threshold indicating a valid touch has takenplace. The rising slope of the signal is steep, as is the falling edge,and it peaks between the First Threshold and the Second Threshold (theconditions for a “tap” selection). FIG. 7B illustrates the signal thatmeets the conditions for a resting set-down action. In this case, therising edge of the touch signal is relatively slow (as compared to a tapsignal) and the amplitude of the signal stabilizes between the First andSecond Thresholds. FIG. 7C illustrates the signal that meets theconditions for a set-down press action. In this case, the rising edge ofthe touch signal is relatively slow as compared to the tap signal, butsimilar in slope to the rising edge of a rest set-down action. However,the amplitude of the signal continues beyond the Second Thresholdindicating the user has pressed harder than a normal touch. The slowerrise time, but higher amplitude indicates a set-down pressing action hastaken place.

Being able to distinguish between a tap selection, a set-down restingaction, and a pressing action is critical in allowing the user to resttheir fingers on a touch surface. Further, using the same sensors todetect all three actions has the advantages of keeping the cost of thesystem relatively lower and simpler.

FIG. 8 shows a flow chart of an exemplary process 800 that allows thesame physical area on a touchscreen keyboard to be used to perform thefunctions of both a numeric keypad and touchpad. The process 800 is notintended to fully detail all the software in its entirety, but isprovided as an overview and an enabling disclosure.

The process 800 is provided by the User Gesture Recognition Software155. At block 805, when the process is first started, various systemvariables are initialized. For example, event time out (threshold time)is set to zero. At block 810, the process waits to be notified that usercontact has occurred within the common area. While the system is waitingin block 810, a counter is incremented with the passage of time. Onceuser contact has occurred, block 815 determines if the counter hasexceeded the maximum time (threshold) allowed for user input (stored asa user option in Data Memory 160).

If the maximum time allowed for user input has been exceeded, then thesystem resets the mode of the common area to the default mode in block820. At a decision block 825, the processor 110 determines whether ornot the current mode is in touchpad mode. If the current mode is in thetouchpad mode, the processor 110 interprets the user contact as atouchpad event and outputs the command accordingly in block 830.

If the current mode is not in the touchpad mode, then the processor 110assumes the common area is in number pad (numpad) mode and proceeds todecision block 835. In touchpad operation, the user will make an initialtouch followed by a sliding motion with their finger (or multiplefingers). In numpad operation, the user will tap on a number key andtypically will not slide their finger. The processor 110 uses thisdifference in typical operation to interpret the user's input indecision block 835 and if a touch-and-slide motion is detected by theprocessor 110 based on signals provided by the sensors (e.g., touchsensors 120 and vibration sensors 125), the processor 110 changes thecurrent mode to the touchpad mode in block 840, and outputs the useraction as a touchpad event in block 845. If the user action is not atouch-and-slide motion then the user action is output by the processor110 as a numpad event in block 850. After blocks 830, 845, 850, theprocess 800 returns to block 810.

Note that single taps (or multiple taps in succession) are also commonwhen using a touchpad, and are commonly assigned to functions such as“select” or what is commonly referred to as a “mouse left button”action. These types of actions typically occur shortly after atouch-and-slide motion, and so the system will still be in touchpad mode(since the counter will not yet have reached the threshold in block815).

Other user gestures on the touchpad are interpreted and assigned tofunctions, such as multiple finger swipes across the touchpad. While thedevice 100 is in the touchpad mode, all these gestures are interpretedas touchpad input and sent to the device's operating system as such tobe interpreted by whatever system software resides therein. In this way,the system and method acts exactly like any other touchpad when intouchpad mode.

In one embodiment, the default mode is set by the user (typicallythrough control panel software). If the device 100 is at rest with nouser input for the user-settable amount of time (threshold), the mode isrestored to the default mode.

Gesture Recognition

Auto-detect mode includes, but is not limited to, recognition of thefollowing gestures and their assigned actions:

Gesture Description Action Single-touch slide The user slides one fingerMouse cursor movement across the surface in touchpad mode. Single-touchtap immediately The user taps their finger on Left mouse button clickfollowing a single-touch slide the surface within a short period of timeafter sliding their finger across the surface Single-touch double-tapThe user taps their finger Left mouse double-click. twice in quicksuccession on the surface within a short period of time after slidingtheir finger across the surface Dual-touch tap immediately The user tapstwo fingers on Right mouse button click following a single-touch slidethe surface within a short (first threshold) period of time aftersliding their finger across the surface Single-touch tap The user tapsthe surface in a Numpad key press single location after not havingtouched it for a long (second threshold) period of time Single-touch tapfollowing The user taps the surface in a Numpad key press anothersingle-touch tap single location within a short period of time afteralso tapping a single location Single-touch tap followed by The usertaps the surface in a Left mouse button press and a single-touch presssingle location immediately drag followed by a touch and hold on thesurface, then drags their finger across the surface

SWYPE Integration

In another embodiment, the touch surface is used in a fourth mode:SWYPE® keyboard. In this mode, the surface represents a keyboard, onwhich the user may slide their finger from letter to letter,implementing the SWYPE paradigm. This mode is manually selected by theuser through some scheme implemented on the keyboard or computersoftware, or it is selected by functionality provided by the auto-detectmode. In the auto-detect mode, the system observes a sliding motionacross the surface and initially interprets it as touchpad movement.However, if the pattern traces out a legitimate word in the fourth mode,the system intelligently switches into the fourth mode and outputs thetext. The system stays in the fourth mode for as long as the user istyping. To exit the fourth mode and return to touchpad mode, the userperforms a gesture—such as pressing and holding their finger for a fewseconds in the same location. Other gestures could also be recognized.

Keyboard Integration

In another aspect of the system, the touch surface is used in a fifthmode: regular keyboard. In the fifth mode, the surface is reconfiguredto be a standard QWERTY keyboard. Using patent-pending touch-taptechnology, the user can rest their fingers on the touch-sensitivesurface and select keys by “tapping” on them. Because this requires morespace than any of the other paradigms listed above, it is possible thatthe device could be used with a diagonal orientation for the fingers. Inother words, fingers are displaced along the axis connection oppositecorners of the surface. Then, the relative displacement of the fingersfrom this resting position can be detected to determine which keys wereselected (as shown in FIG. 9B). In other words, full sized keys wouldnot be necessary, thus saving space yet allowing comfortable typing.

FIG. 9A shows is a schematic of a keyboard used to illustrate a touch-and tap-sensitive keyboard 900 that incorporates on its forward-facingsurface an area 910 incorporating the functions of both a numeric keypadand touchpad. The term “keyboard” in this application refers to anykeyboard that is implemented on a touch- and tap-sensitive surface,including a keyboard presented on a touch-sensitive display. Thekeyboard 900 includes the outline of the area 910 incorporating thefunctions of the touchpad, the keys assigned to the numeric keypad, aswell as the selection keys commonly referred to as the “left and rightmouse buttons” 930. “Mode” refers to the type of function that isassigned to the commonly-shared area 910. A separate mode key 924 allowsthe user to manually select between Touchpad mode, numeric keypad (or“numpad”) mode, or “Auto” mode (whereby the function assigned to commonarea 910 is determined by the system according to the actions of theuser on the surface of the common area 910).

In one embodiment, the system displays the current mode (touchpad ornumber pad) with visual indicators 920 along with an “Auto” mode visualindicator. In this way, the user can know which mode the system is in atall times. In one embodiment, a mode key 924 is provided below theindicators 920 on the keyboard. User activation of the mode key 924causes the processor 110 to switch to another mode.

In one embodiment, the user may define the default mode to be thetouchpad mode by first selecting Auto mode with the mode key 924immediately followed by a touch-and-slide motion on the common area 910.In the absence of a touch-and-slide motion immediately following theselection of Auto mode, the processor 110 will set the default mode tonumpad mode.

In another embodiment, the touch surface is used in a fourth mode:keyboard. In the fourth mode, the surface represents a keyboard, onwhich the user may enter text using a plethora of methods designed forsmaller touch surfaces (such as those invented for smartphones). Thismode is manually selected by the user through some scheme implemented onthe keyboard or computer software, or it is selected by functionalityprovided by the auto-detect mode. The device stays in keyboard mode foras long as the user is typing. To exit the keyboard mode and return tothe touchpad mode, the user performs a predefined gesture—such aspressing and holding all their fingers for a few seconds in the samelocation. The processor recognizes the unique gesture, then changes modeaccordingly. Other gestures could also be recognized.

In another embodiment, the touch surface incorporates a dynamic display.The display changes in accordance with the current mode setting todisplay the appropriate image in the common area. For example, whennumpad mode is selected, a numeric keypad is displayed; when touchpad isselected, a blank rounded rectangle is displayed; and so on.

FIGS. 10A-10F show an exemplary process performed by the device 100. Theflowcharts shown in FIGS. 10A-10F are not intended to fully detail thesoftware in its entirety, but are used for illustrative purposes.

FIG. 10A shows a process 1000 executed by the processor 110 based oninstructions provided by the OSK software component 154. At block 1006,when the process 1000 is first started, various system variables areinitialized, such as minimum rest time, number of finger touchthreshold, drift distance threshold, and key threshold. At block 1008,the process 1000 waits to be notified that a contact has occurred withinthe area of a touch-screen. Then, at block 1010, home-row detectionoccurs based on signals from one or more of the sensors (e.g., touchsensors 120 and vibration sensors 125). Home-row detection is describedin more detail in FIG. 10B. At a block 1012, locations of keys for theto-be-displayed virtual keyboard are determined based on the sensorsignals. The key location determination is described in more detail inFIG. 10C. Next, at block 1016, key activations are processed (see FIGS.10D and 10E for more detail.) At a block 1018, user's finger drift isdetected based on the sensor signals. Finger drift is described in moredetail in FIG. 10F. Then, at block 1020, a virtual keyboard is presentedon the display 130 based on at least one of the determinations made atblocks 1010-1018. The process 1000 repeats when a user removes theireight fingers and then makes contact with the touchscreen.

FIG. 10B shows the home-row detection process 1010. At a decision block1034, the process 1010 determines if a user has rested their fingers onthe touch-screen for a minimum amount of time (i.e., minimum restthreshold). At a decision block 1036, the process 1010 determines if theappropriate number of fingers have rested on the touch surface, thusinitiating a home-row definition event. If the conditions in either ofblocks 1034 or 1036 are not met, the process 1010 exits without changingthe location of the onscreen keyboard.

Once both the time and number of resting fingers requirements are met,the processor 110 determines the location of the resting fingers, seeblock 1040. A KeySpacelndex (or “KSI”) value is then determined in block1042. The KSI is used to customize the onscreen keyboard to the size andspacing of the user's fingers.

The KSI may change from one home-row definition event to the next, evenfor the same user. In one embodiment, all four fingers of each hand areresting on the touch surface to initiate the home-row definition event.In such a case, the KSI is given by the following formula:

KSI=(Average RestingKey Spacing)/(Modeled NominalSpacing)=[(a+b+c)/3]/A=(a+b+c)/3A

where,

-   -   A=a modeled nominal distance between keys (typically 19 mm)    -   a=the measured distance between RestingKey1 and RestingKey2    -   b=distance between RestingKey2 and RestingKey3    -   c=distance between RestingKey3 and RestingKey4.

The KSI formula can be adjusted accordingly if fewer than four restingfingers are used to initiate a home-row definition event (as defined ina set of user preferences stored in a database). The KSI is used insubsequent processes.

A data model for a standard onscreen keyboard is stored in memory of thesystem. In this data model, the onscreen keyboard layout is divided intotwo sections: keys normally typed with the right hand, and keys normallytyped with the left hand. Further, each key is related to the home-rowresting key that is rested upon by the finger that is most likely totype that particular key (defined as the “related resting key”). Thelocation of each key is defined in the data model as a relativemeasurement from its related resting key.

An exemplary formula for determining the location of each key is givenas:

Key(x′, y′)=KeyModel (x*KSI, y*KSI)

Where,

-   -   x=the nominal stored x distance from the center of the Related        Resting Key (RRK)    -   y=the nominal stored y distance from the center of the RRK

It is possible that the modified key positions of two or more keys mayoverlap. If that is the case, the size of the overlapping keys isreduced until the overlap is eliminated.

The orientation of the X-Y axis is determined separately for eachresting key. For each of the left and right sectors, a curve is fit tothe resting keys in that sector. The X-Y axis for each key is thenoriented to be the tangent (for the x-axis) and orthogonal-tangent (forthe y-axis) to the curve at the center of that key.

FIG. 10C shows the assigning key locations process 1012. The process1012 is repeated for each key of the keyboard. At block 1052, apre-stored location for each key is retrieved from the database 181,relative to its associated resting key position in the form [RestingKey,Δx, Δy]. For example, the key representing the letter “R” is associatedwith the resting key L1 (typically the letter “F”), and is positioned upand to the left of L1. Thus, its data set would be [L1, −5, 19] (asmeasured in millimeters). Similar data is retrieved for each key fromthe database 181. At block 1054, a new relative offset is calculated foreach key by multiplying the offset retrieved from the database by theKSI. At block 1058, the absolute coordinates of each key is thendetermined by adding the new offset to the absolute location of theassociated resting key as determined at block 1054. At decision block1060, the process 1012 tests to see if any keys are overlapping, and ifso, their size and location are adjusted at block 1062 to eliminate anyoverlap. Then the process 1012 returns to the process 1000.

FIG. 10D shows the process-key actuations process 1016, whereby theactual key events are determined and output. The process 1016 begins atdecision block 1070 that tests if a valid touch-tap event has occurred.This is determined through a correlation between the touch sensor(s) 120and vibration sensor(s) 125. Candidate keys are scored by applying a keyscoring algorithm at block 1072. The key with the highest score is thenoutput at block 1074 and the process 1016 returns.

FIG. 10E shows a process for the key scoring algorithm from block 1072of FIG. 10D. At block 1080, signals received by the touch sensors 120and the vibration sensors 125 are correlated to determine where theuser's tap took place and to define keys in the immediate vicinity as“candidate keys”. By considering keys surrounding the area of the tap(rather than just the key where the tap took place), the processor 110accounts for ambiguity in the user's typing style. At a decision block1082, the process 1072 tests to see if the user moved their finger froma resting key to type. Note that in typical typing styles, even a10-finger touch typist will not constantly rest all four fingers at alltimes. So, it isn't a prerequisite that a change in a resting key takeplace in order for a valid key to be typed on. However, if a change doestake place to the state of a resting key in the vicinity of thecandidate keys (or if it is a candidate key itself), useful informationcan be obtained from such change as explained at block 1084. At block1084 a virtual line is calculated between the resting key in thevicinity of the tap for which a state change was detected, and thelocation of the tap, as calculated at block 1080. The virtual lineextends beyond the tap location. At block 1084 keys that the projectedline passes through or by are determined and the processor 110 increasesthe score of those keys accordingly. In this way, relative movements inthe direction of the desired key are correlated to that key, even if thetap location doesn't occur directly on the key. At block 1088, theprocessor 110 takes into account the preceding words and characters thatwere typed as compared with linguistic data stored in data memory 181(e.g., statistical databases 181). This includes commonly knowndisambiguation methods such as: letter-pair statistical frequencies,partial-match prediction, inter-word prediction, and intra-wordprediction. Appropriate scoring is assigned to each candidate key. Atblock 1090, the candidate key with the highest score representing thehighest calculated probability of the user's intended selection isdetermined and the process 1072 returns.

FIG. 1OF shows the drift detection process 1018 for accommodating whenthe user inadvertently moves their hands (or “drifting”) as they type.The process 1018, at block 1091, compares the actual tap location withthe current center of the displayed intended key, and stores thedifference in the X and Y coordinates ΔX and ΔY. These differences areadded to a previous cumulative total from previous keystrokes at block1092. At decision block 1093, the processor 110 tests if the cumulativedifference in either direction exceeds a pre-stored variable called“DriftThreshold” (as defined from user preference or default data storedin data memory 162 (e.g., user options/preferences 162)). If thethreshold is exceeded, the processor 110 moves the location of theentire keyboard in block 1094 by the average of all ΔXs and all ΔYssince the last location definition event. If the cumulative differencesdo not exceed the DriftThreshold for the entire keyboard, then a similarcalculation for the individual selected key is performed at block 1097.At decision block 1098, the processor 110 tests if the cumulativedifferences for that individual key exceeds the user-defined keythreshold after block 1097 and, if so, adjusts its location at block1099. The key threshold is the permissible amount of error in thelocation of the tap as compared to the current location of theassociated key. When key threshold has been exceeded, the associated keywill be moved. After block 1094, if the decision at block 1098 is No, orafter block 1099, then at block 1095, the processor 110 tests if any ofthe new positions overlap with any other keys and if the overallkeyboard is still within the boundaries of the touch sensors. If thereare any conflicts for either test, they are corrected with a “best fit”algorithm in block 1096 and then exits. Also, if no conflicts are found,the process 1018 returns.

Even though the method will allow the user to type without the onscreenkeyboard being visible, there are still times when a user will want toview the keys. For example, if they don't know which key is associatedwith a desired character, or where certain characters are located on aseparate numeric and/or symbols layer. Other users may not be able totype from rote, knowing by memory where each character is located. Forthese, and other reasons, it is important to visually present theonscreen keyboard on the screen of the device.

According to stored user's preference, the onscreen keyboard can remainvisible continuously while typing is taking place. Alternatively, theonscreen keyboard becomes transparent after the home-row definitionevent. In one embodiment, the onscreen keyboard becomes semitransparentso-as to allow the user to see through the keyboard to content on thescreen below.

In the case where the keyboard is set to be invisible, other content maybe displayed on the full screen. There may be other user interfaceelements, such as buttons, that will appear to be active yet be locatedbelow the invisible onscreen keyboard. In such a case, the device 100intercepts the user's input directed toward such an element and causesthe onscreen keyboard to become visible, reminding the user that it isindeed present. The user may then elect to “put away” the keyboard bypressing a corresponding key on the keyboard. Note that putting away thekeyboard is not the same as making it invisible. Putting away thekeyboard means to “minimize” it off the screen altogether, as is acommon practice on touchscreen devices.

In one embodiment, the onscreen keyboard cycles between visible andinvisible as the user types. Each time the user taps on the “hidden”onscreen keyboard, the onscreen keyboard temporarily appears and thenfades away after a user-settable amount of time.

In one embodiment, only certain keys become visible after eachkeystroke. The keys that become temporarily visible are those keys thatare most likely to follow the immediately preceding text input sequence(as determined based on word and letter databases stored in the system).

In one embodiment, the onscreen keyboard becomes temporarily visiblewhen the user, with fingers resting in the home-row position, pressesdown on the surface with their resting fingers based on changes sensedby the touch sensors 120.

In one embodiment, the onscreen keyboard becomes visible when the userperforms a predefined action on the edge of the enclosure outside of thetouch sensor area, such as a double- or triple-tap.

The onscreen keyboard, if set to appear, will typically do so when atext-insertion condition exists (as indicated by the operating system151), commonly represented visually by an insertion carat (or similarindicator).

In one embodiment, the tactile markers commonly used on the F and Jhome-row keys are simulated by providing haptic feedback (such as avibration induced on the touchscreen) when the user positions theirfingers to rest on those keys. In this way, the user may choose for thekeyboard to remain stationary in the same onscreen position, yet findthe correct placement of their hands by touch only (without looking).

To increase the accuracy of the keyboard, statistical models of languageare used. If a touch/tap event yields an ambiguous key choice, thestatistical models are called upon by the processor 110 to offer the keythat is most likely what the user intended.

This “disambiguation” is different from other methods used for othertext input systems because a permanent decision about the desired keymust be made on the fly. There is no end-of-word delineation from whichword choices can be displayed to the user and the output modified.Instead, each time the user taps on a key, a decision must be made and akey actuation must be sent to a target application program (i.e., textentry program).

Several statistical analysis methods can be employed: partial-matchletter prediction, current-word prediction, next-word prediction, andconjunctive next-word prediction. These are explained in detail in thefollowing sections.

Prediction by Partial Match

A well-known algorithm originally invented for data compression usefulin this case is prediction by partial match (or PPM). Applied to akeyboard, the PPM algorithm is used to predict the most likely nextcharacter, given a string of characters that has already occurred (oflength k). Computing time and resources grow exponential with the valueof k. Therefore, it is best to use the lowest value of k that stillyields acceptable disambiguation results.

By way of example, let k=2. A process looks back at the past twocharacters that have been entered and then compare probabilities from adatabase of the most likely next character(s) to be typed. For example,the underlined letters below show what is used to predict the next mostlikely letter:

An

An

An e

An ex

An exa

An exam

An examp

An exampl

An example

The data storage required for this algorithm for a total number ofpossible keys A is: A^(k+1)

For a typical onscreen keyboard, this process consumes less than 1 MB ofdata.

The statistical model is built up for each language (although with asmall value for k); the table may be similar for languages with commonroots. The model also dynamically updates as the user enters text. Inthis way, the system learns the users typing patterns and moreaccurately predicts them as time goes on.

Language variants are provided in the form of language-specificdictionaries configured through an operating system control panel. Thecontrol panel identifies the current user's language from the systemlocale and selects the appropriate prediction dictionary. The dictionaryis queried using a continuously running “systray” application that alsoprovides new word identification and common word usage scoring.

In one embodiment, a database made up of commonly used words in alanguage is used to disambiguate intended key actuations. The algorithmsimply compares the letters typed thus far with a word database, andthen predicts the most likely next letter based on matches in thedatabase.

For example, say the user has typed “Hel.” Possible matches in the worddatabase are:

Hello (50)

Help (20)

Hell (15)

Helicopter (10)

Hellacious (5)

The numbers beside each word represent their “frequency” of use,normalized to 100. (For convenience sake, the total frequencies in thisexample add up to 100, but that would not normally be the case).

The candidate letters that most likely follow “Hel” are:

L (70)—probabilities added for the words “Hello”, “Hell”, and“Hellacious”

P (20)

I (20)

This example is particularly useful, in that the letters L, P, and I areall in close proximity to one another. It is possible, and even likely,that the user may tap on a location that is ambiguously near severalkeys (I, 0, P, or L, for example). By adding word prediction, the choiceis significantly disambiguated; in this example, the obvious most-likelynext letter is “L.”

Note that this implementation of the word prediction algorithm isdifferent from that traditionally used for onscreen keyboards, becauseit is not truly a word prediction system at all: it is a letterprediction system that uses a word database.

In one embodiment, word pairs are used to further disambiguate the mostlikely selected key. With simple word prediction, there is no context todisambiguate the first letter of the current word; it is completelyambiguous. (This disambiguation is reduced slightly for the secondletter of the word, and so on for the remainder of the word.) Theambiguous nature of the first few letters of a word can be significantlyreduced by taking into account the word that was entered immediatelyprevious to the current word; this is called “next-word prediction”.

For example, if the word just typed was “Cleankeys”, common next wordsstored in the database may be:

Keyboard (80)

Inc. (20)

Is (20)

Will (15)

Makes (10)

Touch (5)

If the user ambiguously taps between the I key and the K key for thestart of the next word, the next-word prediction algorithm can helpdisambiguate (in this case, “K” would win).

Logic may dictate that the concept of considering the previous wordtyped could be carried to the previous k words typed. For example, fork=2, the system could store a database that has 2nd-degree next-words(or next-next-words) for every word in the database. In other words,look back at the two previous words in combination to determine the mostlikely word to follow. However, this quickly becomes unwieldy, both interms of space and computing power. It simply isn't practical to storethat many combinations, nor is it very useful, because most of thosecombinations would never occur.

There is, however, a significant exception that is worth considering:words that have a very large number of next-word candidates. Such is thecase for parts of speech known as conjunctions and articles.

The seven most-used conjunctions in the English language are: and, but,or, for, yet, so, nor.

The articles in the English language are: the, a, an.

By special-casing these 10 words, the system improves first-letterpredictions.

Consider the phrase: kick the.

Because every noun in the database is most likely a next-word candidatefor the article “the”, there is very little use derived from thenext-word prediction algorithm. However, if the context of “kick” beforethe article “the” is retained, a much richer next-next-word choice isattained. Effectively, a new “word” is stored in the database called“kick the.” This new entity has the following next-word candidates:

Ball (50)

Bucket (20)

Habit (15)

Can (10)

Tires (5)

Thus one can confidently predict that the most likely next letter tofollow the phrase “kick the ” is the letter “B.”

Any word that is found combined with a conjunction or article iscombined with those parts of speech to form a new word entity.

A notable difference between the letter-by-letter prediction systemdescribed herein and a word-based prediction system is the ability todynamically reorient the prediction for each letter. For example, if aguess is wrong for a specific key and the desired word subsequentlybecomes clear, the algorithm abandons the choice it made for theincorrect letter and applies predictions for the remaining letters,based on the newly determined target word.

For example:

Ambiguous Text Entered Candidate Keys Predicted Words Predicted LetterKick the B, h, g Ball, bucket, habit, B goat, garage Kick the b A, q, sBall, habit, wage A Kick the ba B, v, space habit B Kick the bab I, k, ohabit I Kick the babi T, r habit T Kick the babit Space, n, m habitspace

As the word progresses, it is shown that the initial letter “B” shouldhave been an “H” (these letters are near one another on the qwertykeyboard layout and one could easily be mistaken for the other). Butrather than commit completely to that first letter, and only considerwords that start with “B,” other candidates are still considered by thesystem in predicting the second letter. So, B, H, and G are consideredas the first letter for subsequent keys. In this way, the mistake isn'tpropagated and the user would need to only make one correction insteadof potentially many.

So, for each new key entered, keys that are adjacent to the new key, aswell as other ambiguous candidates, are considered as possibilities indetermining subsequent letters.

When a mistake is made and the user backspaces and corrects it, thesystem can feed that data back into the algorithm and make adjustmentsaccordingly.

For example, the user ambiguously enters a key in the middle of thekeyboard and the scoring algorithm indicates that potential candidatesare “H,” “J,” and “N”; the scores for those three letter fall into theacceptable range and the best score is taken. In this example, let's saythe algorithm returns the letter “J” as the most likely candidate and sothat is what the keyboard outputs. Immediately following this, the userunambiguously types a <backspace> and then an “H,” thus correcting theerror.

This information is fed back into the scoring algorithm, which looks atwhich sub-algorithms scored an “H” higher than “J” when the ambiguouskey was originally entered. The weighting for those algorithms isincreased so if the same ambiguous input were to happen again, theletter “H” would be chosen. In this way, a feedback loop is providedbased directly on user corrections.

Of course, the user can make typing mistakes themselves that are not theresult of the algorithm; it correctly output what the user typed. So,care must be taken when determining if the user correction feedback loopshould be initiated. It typically occurs only when the key in questionwas ambiguous.

A user-settable option could allow the keyboard to issue backspaces andnew letters to correct a word that was obviously wrong. In the exampleabove, once the predictor determines that the only logical word choiceis “habit,” the keyboard would issue backspaces, change the “b” to an“h,” reissue the subsequent letters (and possibly even complete theword).

With so many factors lending to the disambiguation of a key, allalgorithms can potentially add to the candidacy of a key. This approachis called scoring; all algorithms are weighted and then added together.The weighting is dynamically changed, to tune the scoring algorithm tothe user's typing style and environment.

FIG. 11A shows a schematic view representative of a typical handheldtablet computer 1150 that incorporates on its forward-facing surface atouch-sensitive display 1152 and a keyboard 1154 designed and used inaccordance with an embodiment. The keyboard 1154, when used inaccordance with some embodiments, generates text that is output to thetext display region 1158 at a text insertion location 1160. The term“keyboard” in this application refers to any keyboard that isimplemented on a touch-and tap-sensitive surface, including a keyboardpresented on a touch-sensitive display. The keyboard 1154 shows theletters of the alphabet of the respective language selected by the useron individual keys, arranged in approximately the standard “QWERTY”arrangement found on most keyboards.

In one embodiment, the orientation, location, and size of the keyboard(as well as individual keys) are adaptively changed according to theinput behavior of the user. When the user rests their fingers on thetouch surface 1152 in a certain way, the system moves the keyboard 1154to the location determined by the resting fingers. When the user intendsto actuate a key on the keyboard 1154, they “tap” on the desired key bylifting their finger and striking the surface 1152 with discernibleforce. User taps that occur on areas 1162, 1164 outside of the touchsensor area 1152 are detected by the vibration sensor(s) and may also beassigned to keyboard functions, such as the space bar.

The absence of a touch sensor signal is in effect, a signal with a valueof zero, and when correlated with a tap (or vibration) sensor can beused to uniquely identify a tap location. In one embodiment, thevibration signal for specific regions outside of the touch sensor area1152, such as those indicated at areas 1162, 1164, are unique and storedin a database by the system. When the absence of a touch signal occursin conjunction with a tap event, the system compares the vibrationcharacteristics of the tap with those stored in the database todetermine the location of the external tap. In one embodiment, the lowerouter boundary area 1162 is assigned to a space function, while theright outer boundary area 1164 is assigned to a backspace function.

FIG. 11B is a schematic view representative of an exemplary virtualon-screen keyboard 1170. The keyboard 1170 is divided into two halves: aleft half 1172 and a right half 1174 (as correlates to the left andright hands of the user). The two separate halves 1172, 1174 are notaligned with each other. The eight keys 1178 that are typically restedon by the user are labeled in bold according to which finger istypically used for that key (e.g., L1 represents the index finger of theleft hand, L4 represents the little finger of the left hand, and so on).All other non-home-row keys are indicated by a label showing whichfinger is normally used to type that key using conventional touch-typingtechniques. It should be noted, however, that there are many typingstyles that do not use the finger placements as shown in FIG. 11B, andthose labels are included herein for illustrative purposes only.

The left half of the keyboard 1172 shows all the keys aligned inhorizontal rows, as they would be on a traditional electromechanicalkeyboard. In one embodiment as shown on the right half 1174, thehome-row keys are dispersed along an arc to better fit the normalresting position of the user's four fingers. Non-home-row keys aresimilarly dispersed in accordance with their relative location to thehome-row resting keys. Further, in one embodiment, the size of each keymay also vary in accordance with the statistical likelihood that theuser will select that key (the higher the likelihood, the larger thekey).

FIG. 11C is a schematic view representative of the virtual onscreenkeyboard 1184 that is oriented at an angle in accordance with anembodiment. The user may rest their hands 1190 on the touch-sensitivesurface 1192 of a typical handheld tablet computer 1194 at any locationand orientation that they wish. In this case, the hands are spread apartfurther than normal and oriented at an angle as referenced to thestraight edges of the device 1194. The user initiates an actionindicating a “home-row definition event,” which, may include, but is notlimited to, the following: resting all eight fingers for a short,user-definable period of time; double-tapping all eight fingerssimultaneously on the surface 1192 and then resting them on the surface1192; or pressing down all eight fingers simultaneously as they areresting on the surface 1192. In another embodiment, not all eightfingers are required to initiate a home-row definition event. Forexample, if someone was missing their middle finger, a home-rowdefinition event may be initiated by only three fingers on that hand.Here the user has rested their hands 1190 at an angle on the tabletcomputer 1194, thus causing a processor of the computer 1194 to generateand display the virtual onscreen keyboard 1184 at an angle.

Turning now to FIGS. 12-21, embodiments for detecting hover-based inputsand discussed.

In particular, FIG. 12 provides an overview of the sequentialfunctioning of a system for using hover to improve typing accuracy, inaccordance with some embodiments. It begins by a user approaching atouch-screen device with the intention of interacting with it. In oneexample, the user may wish to type on the screen of the device using anonscreen keyboard. The user begins by resting their fingers on thesurface of the device in an area designated for the onscreen keyboard(1202). The system recognizes this “set-down” gesture and automaticallyforms the keys of the onscreen keyboard beneath the user's fingers, withpreference being given to placing the home keys (F and J) beneath theuser's index fingers. The system also takes note of the physicalcharacteristics of the user's hands—the size and spacing between theirfingers, among other parameters—and stores this information as theuser's “hand print” (1204). In some embodiments, the onscreen keyboardis relocated to an adjusted position beneath the user's resting fingers(1206).

In some embodiments, the user lifts their hands off the surface of thedevice and begins to type. Many users tend to hover their fingersslightly above the keys of the onscreen keyboard as they type,interspersed with occasional rests on the surface. As the user hoverstheir fingers above the surface (1208), the system determines thelocation of each of the user's fingers either directly from the hoversensor data (if it is accurate enough) or it references the user's handprint to interpolate the location of each finger (1210). In someembodiments, the onscreen keyboard is displayed/relocated at updatedpositions according to determined/resolved positions of the user'shovering fingers (1212).

In some embodiment, the system continues to track each of the user'sfingers as they type, hover, set-down, rest, and so on. As keys areselected, the system learns which fingers were used to make saidselections and stores that information in a database (“finger-key”database). In cases when the key selection is ambiguous (unclear whichkey the user intended to select—hitting between two keys, for example),the system refers to the finger-key database to help disambiguate thekey selection. In some embodiments, the system records, as a database ofkey-to-finger associations, user keystrokes directed to respective keystogether with information that indicates which of the user's fingerseffect the keystrokes, the information indicating which of the user'sfingers effected the keystrokes being based, as least in part, on theresolved positions of the user's hover fingers (1214).

If, for example, the user ambiguously touches the onscreen keyboardbetween the “i” and “o” keys with their middle finger (1216), the systemrefers to the finger-key database to determine that the user most oftenselects the letter “i” with their middle finger (and “o” with their ringfinger) (1218). So in this example, the system would significantlyincrease the probability that the intended key of the ambiguousselection is the letter “i”. Thus, the hover detection and fingertracking system can greatly enhance the accuracy of a user's typingexperience

Some embodiments also track a location of each individual finger of theuser. FIG. 13A is a schematic view of a portable computer with atouch-screen display on which a keyboard system of the present inventionis displayed. This illustrates the ability for a user to rest theirhands on the surface at any angle. Using the system and methods of thepresent invention, the keyboard is able to form around the user'sfingers. In some embodiments, the user is able to manually determine thelocation of the keyboard by setting down at least three fingers on eachhand and resting them on the touch surface. This is defined as a“set-down event.” During a set-down event, the user may drag the twohalves of the keyboard independently. The location of each half of thekeyboard is determined by locating the home keys (F and J) directlybeneath the user's index fingers of the left and right handsrespectively. Since a three-finger set-down is permitted, the systemmust still be able to identify the location of each finger—even if theyare not all present.

In some embodiments, the system requires the user to simultaneously resteither 3 or 4 fingers per hand to define a “set-down event”. As oneexample, FIGS. 15A-15F show different three and four finger combinationsthat can result in a set-down event. The fingers are numbered asfollows: 1 (thumb), 2 (index), 3 (middle), 4 (ring), and 5 (littlefinger). In the instance when the system is implemented on a solidtouch-sensitive surface (such as a touchscreen on a tablet computer), itmay be difficult to provide the user with a tactile marker on the homekeys (i.e. keys in home row positions of a keyboard) in order to guide auser to a proper “set-down event.” Instead, the system may move the homerow under the user's fingers. It is not necessary for all fingers to betouching the surface concurrently in order to determine the home rowposition. The location of the home row is dictated by the location ofthe “home keys” F and J. In some embodiments, locations of the left andright home rows, and thus keyboard halves, are determined independently.Even if the index finger (typically associated with the home keys) isnot resting, its approximate location can be determined using the otherfingers that are touching and applying known metrics of human anatomyderived from the aforementioned hand metrics database. In someembodiments, a home-row definition event may take place by either theuser resting their fingers on the surface for a prescribed amount oftime, or by the user hovering over the keys with their fingers for aprescribed amount of time. In both cases, the location, size, andorientation of the onscreen keyboard is determined according to thelocation of the user's fingers.

FIG. 13A also illustrates a typical user typing on the touch-screen of amobile device. Note how the hands don't have to be aligned with theedges of the device; rather, the onscreen keyboard appears at whateverangle the hands are placed. On smaller devices, this provides theadvantage of more screen space for the user's hands when placed on adiagonal setting as shown in FIG. 13A.

FIG. 13B illustrates the signals received by the touch sensors of thetouch surface rested on by the user's hands in FIG. 13A. The darker theshape, the stronger the touch signal received. FIG. 13B suggests thatthe left thumb and right palm are lightly resting on the surface, whilethe other digits are in firm contact with the surface. Thus, in theexample illustrated in FIG. 13B, the user is firmly resting all eightfingers, but their left thumb is hovering just about the surfaceresulting in a weaker touch signal. The same is true for the user'sright palm.

Some embodiments are also used to determine a location of a user'sfinger relative to a touch-sensitive display. The location of a user'sfingers can be determined in a number of ways. In one embodiment, thelocation of the user's fingers is determined by a touch sensor (or arrayof touch sensors) as the user rests their fingers on a touch-sensitivesurface. Information such as hand anatomy, the size and relativelocations of finger touches, the strength of the finger set-down, andthe weight of each finger can all provide clues as to accuratelydetermine which finger is which. In this case, the touch sensor may becapacitive, force sensing, optical, or any number of commonly used touchsensing methodologies.

In some embodiments, the location of the user's fingers is determined bya hover sensor (also referred to as a proximity sensor) as the user'sfingers hover above a touch-sensitive surface. The hover sensor detectsthe presence of human hands and fingers a reasonable resting distanceabove the touch surface. The hover sensor may be capacitive, optical, orany number of commonly used proximity sensing methodologies. In someembodiments, the location of the user's fingers is determined both by acombination of the user resting on and hovering their fingers above thetouch surface.

Many touch capacitive systems are now capable of projecting their fieldsfar enough above the surface that capacitive objects in close proximitycan be sensed without the objects actually touching the surface. Thesame can be said for other types of touch sensors, including (but notlimited to) imaging, optical, infrared, temperature, thermal imaging,and standing acoustical wave (SAW) sensors.

In some embodiments, the touch sensor is a capacitive sensor capable ofprojecting its sensing a spatial region well above the touch surface,and is thus sufficient to detect the presence of a user's fingershovering above the surface: both individually and as a group. It iscommon for the accuracy of the capacitive touch sensor to decrease asthe user's fingers move further from the touch surface. In some cases,the capacitive sensors may not be able to discern each individualfinger, but instead senses only a general area comprised of all thefingers of a hand. In this case, the system may apply known factors ofhuman anatomy and the relative size/location of each finger in order toestimate the location of each finger.

In some embodiments, other types of sensors may be used to accuratelydetermine the location of each individual finger, including, but notlimited to: imaging, optical (e.g., a camera point in such a way thatits field of view includes a user's hand and fingers), infrared,temperature, thermal imaging, and standing acoustical wave (SAW)sensors. In some embodiments, image recognition (to accurately detectfinger locations) is improved by using an infrared light emitter andinfrared detection camera. In these embodiments, the light illuminatesnear objects more brightly than objects that are farther away. In thisway, the user's hands and fingers are more easily recognized. A commonexample of this type of technology is the “Kinnect” camera made popularon Microsoft's Xbox platform. In other embodiments, ultrasonic soundwaves are emitted from the mobile device and then an array ofmicrophones (at least 3) is used to measure the reflection of the soundswaves. In this way, the system is able to accurately detect objectswithin a certain detection zone of the ultrasonic sound waves.

FIG. 14A is an illustration depicting a user's fingers and thumbsresting uniformly on a touch surface. In some embodiments, a user mayrest their fingers in a non-uniform way (as shown in FIG. 14B). Forexample, it is often the case that the user may curl their fingers, liftsome higher than others, actually be missing fingers, or a myriad ofother circumstances that results in not all ten digits being detectableat any given time by the system. A depiction of an example of a touchsensor output that is produced based on proximity of a user's fingers toa touch screen (e.g., directly contacting or hovering above the touchscreen) is shown in FIG. 14A. The four fingers and thumb of the lefthand are accurately shown as 1301 in FIG. 14A. The right hand indexfinger is shown as 1302 and a smaller touch signature is shown as thelittle finger at 1303. The opposed thumb is shown at a slightlydifferent angle as it rests on the touch surface at 1304.

FIG. 14B illustrates how the left hand at 1401 registers only 3 fingers.The middle and ring fingers of the right hand are shown overlapping at1402. A large touch region caused by the user resting their right palmon the touchscreen is depicted at 1403. The system must be able todiscern these anomalies in order to correlate touch locations tofingers.

Image recognition algorithms may be applied to each of the registeredtouch locations. Parameters that may be used to determine the mapping offingers to the sensed touch regions include:

-   -   Area of the touch region    -   Shape of the touch region    -   Strength of the touch signal    -   Relative location of the touch region to other touch regions    -   The absolute location of the touch region on the touch surface    -   The location of the touch region in relation to the onscreen        keyboard (and keys in that area that have specific finger        mappings)    -   Historical placement of previously determined fingers on or near        the location of the touch region    -   Horizontal placement of the touch region on the touch sensor    -   Vertical placement of the touch region on the touch sensor    -   Other parameters specific to the touch region

In some embodiments, fingers are associated with finger identifiers inorder to help map fingers to keys. FIG. 14C shows one example mapping offingers to finger identifiers, in order to help build finger-to-keyencodings in accordance with some embodiments. As shown in FIG. 14C,each finger is mapped to a particular finger identifier left pinky toL5, left ringer finger to L4, left thumb to L1, and etc. (and the samefor the right hand as well).

In some embodiments, once the fingers are located (e.g., using thetechniques discussed herein, such as hover tracking before contact),they can be tracked and correlated with touches/taps on the touchsurface of mobile devices associated with key selections on an onscreenkeyboard. Each finger is assigned a unique code or identifier (asdiscussed above in reference to FIG. 14C), and the system learns (or istrained) on which finger the user typically uses to select each key onan onscreen keyboard. Then each word the user types can be encoded bywhich finger was used to select each letter in order.

For example, suppose a user was a trained 10-finger typist and typed theword “this”. The resulting finger-code would be: L2 (t), R2 (h), R3 (i),L4 (s). For most words, these codes are unique. One benefit is thatusers can be sloppy about where they press a key; they simply touch withthe correct finger in the general area and the algorithm will match thesequence to the most likely word in the language model. No need to worryabout directing the touch to a specific row of the onscreen keyboard.

In this way, the concept of key locations becomes redundant, and thismethod could be implemented on a surface with no onscreen keyboardvisible at all. (The method is particularly helpful to typists who havedeveloped at least some level of typing muscle memory where they knowwhere each key is on the keyboard without looking).

The encoding of words can be pre-stored for common typing styles(10-finger, etc.), but the system can also learn the codes specific toeach user as they type. The user could reduce the amount of time neededfor the system to learn their typing style by undergoing a trainingsession (such as typing “the quick brown fox jumps over the lazy dog”,which includes every letter on the keyboard).

Shorter words will possibly have ambiguity. For example, the words fan,van, ban, ran, and tan all have the same finger coding for a 10-fingertypist: L2, L5, R2. A Language Model could be used to help disambiguatesuch cases, by taking into consideration the frequency of use, bi-grams,tri-grams, context of the text, and many other metrics used in languagemodels.

The location of each keystroke relative to a baseline of where the userrests each finger can be used to help disambiguate the word (+y, −y, +x,−x). In the above example, the words could be encoded as follows:

Fan: L2, L5, R2(−x, −y)

Van: L2 (−y), L5, R2 (−x, −y)

Ban: L2 (−y, +x), L5, R2 (−x, −y)

Ran: L2 (+y), L5, R2 (−x, −y)

Tan: L2 (+y, +x), L5, R2 (−x, −y)

The system detects even slight changes in location for each finger, soit is unnecessary for the user to move their fingers the full distancefrom row to row, or key to key.

A database of encoded words can be created for each individual user(either locally on the mobile device, or in the cloud). A user can beidentified by their unique typing characteristics. Once a user isidentified, their database is loaded by the system. As one example, fora 10-finger trained typist the finger-to-key encodings for the words“HELLO” and “WORLD” may be as shown in tables 1 and 2, respectively,below:

TABLE 1 finger-to-key encodings for typing of the word “hello” LetterFinger code H R2 E L3 L R4 L R4 O R4

TABLE 2 finger-to-key encodings for typing of the word “world” LetterFinger code W L4 O R4 R L2 L R4 D L3

FIGS. 15A through 15F illustrate different combinations in which threeor four fingers can be set down on an onscreen keyboard. The systemidentifies each finger in all cases based on the relative location ofeach touch point. It does this from a database containing human handanatomy information.

FIGS. 16A through 16G show a method (e.g., implemented as a softwarealgorithm executable by a computing device with one or more processorsand a touch-sensitive display) to determine the location of an onscreenkeyboard, identify and track the user's fingers, and store associationsbetween fingers and keys selected.

In particular, FIG. 16A is a high-level overview of what steps thesystem takes when actual touches are detected on the touch surface. Insome embodiments, touches are detected (6100), assigned to either leftor right hand (6105), assigned to particular fingers (6110), fingers arethen tracked (6115), status is reported to a state machine (6120), andthen based on whether the fingers are lifted, the method eithercontinues from 6100 (6125-Y) or continues from 6115 (6125-N). In someembodiments, these finger tracking techniques are used to buildrepositories, such as those discussed below in reference to FIG. 16H. Insome embodiments, the finger tracking techniques are further used todetermine fingers that were used to type keys (e.g., operations 1683 and1684, FIG. 16H).

FIG. 16B describes an algorithm in which a commonly-known “K-Means”filter is applied iteratively to divide the touches into three regions:left, right, and middle. Logic and knowledge of human hand anatomy isthen applied to assign the middle touches to either the left or righthands. The result of this step is all touches are assigned to either theleft or right hand. In some embodiments, FIG. 16B provides more detailsfor operation 6105 of FIG. 16A. As shown in FIG. 16B, assigning touchesto a left or right hand (6200) includes determining (6205) a totalnumber of touches (e.g., keystrokes at a virtual keyboard); detecting(6210) presence of a thumb based on low Y-values and removing datarelated to the thumb from further analysis (in some embodiments,location and direction of a thumb contact with a touch-sensitive surfaceis used to help determine whether a left or right hand was used); andassigning (6215) 3 K-Means centroids to divide the screen in 4 equalquarters in the X-Direction and centered vertically on theonscreen/virtual keyboard. The algorithm or method also includesexecuting (6220) K-means filters iteratively until 3 distinct clustersexist (left, right, and middle); assigning (6225) left cluster to lefthand and right cluster to right hand; and determining (6230) how manytouches are in the middle cluster. If 4 touches are determined to be inthe middle cluster, then the left-most touches are assigned to the leftcluster, right-most touches are assigned to the right cluster, and theanalysis proceeds to 2 points evaluation in operation 6250 (6235). If 3touches are determined to be in the middle cluster, then the pointfarthest from center is assigned to its respective side and the analysisproceeds to 2 points evaluation in operation 6250 (6240). If 1 touch isdetermined to be in the middle cluster, then the touches are assigned tothe side with the fewest number of touches (6245) and the analysisproceeds to operation 6255. If 0 touches are determined to be in themiddle cluster, then the analysis proceeds directly to operation 6255.

The 2 points evaluation (6250) includes assigning the point with thehigher Y value to the left or right cluster with highest Y average, andvice versa, the analysis then proceeds to operation 6255. At operation6255, the results are evaluated to determine whether they comply withhand anatomy parameters (6255). If yes (6255-Y), then the algorithmreturns (6265). If no (6255-N), then the K-means centroids are adjustedin order to better assign touches so that the assignments will bettermatch the hand anatomy parameters.

FIG. 16C is a high-level overview of the steps taken by the system toassign touches to specific fingers of the user. In some embodiments,either three or four concurrent touches are required for a validset-down event. So the algorithm described in FIG. 16C considers onlythose two cases. In some embodiments, FIG. 16C provides additionaldetails regarding operation 6110 (FIG. 16A). As shown in FIG. 16C,assigning fingers to touches (6300) includes determining (6305) how manytouches are assigned to a hand (e.g., as determined by the algorithmdescribed above in reference to FIG. 16B. If 3 (6305-3), then thecomplex deltas algorithm (6310) and simple deltas algorithm (6315) areexecuted. If 4 (6305-4), then all four touches are assigned torespective fingers of that hand (6320). At 6325, the method returns.

FIG. 16D describes a robust algorithm titled “Complex Deltas.” Thealgorithm calculates the relative distance ratios between points definedas dRatio=(Distance between a pair of points)/(Total Distance) (atoperation 6405) and also calculates the angle formed between each pairof points. These values are compared with “hand model” databasecontaining parameters of typical human hand anatomy. An example of thisdata is shown in FIG. 19. The result of the Complex Deltas algorithm isthe assignment of a specific finger to each touch point. In particular,if the dRatio is not greater than 25% (6410-No), then at 6430 thealgorithm determines that fingers 2 and 5 are not both down. Next, thealgorithm calculates relative adjacent angles (6435) and compares theresults with a hand anatomy database to determine either (2, 3, 4) or(3, 4, 5) fingers are down (6440).

In some embodiments, if the dRatio is greater than 25% (6410-Yes), thenat 6415 the algorithm determines that fingers 2 and 5 are both down.Next, the algorithm calculates relative adjacent angles (6420) andcompares the results with a hand anatomy database to determine either(2, 3, 5) or (2, 4, 5) fingers are down (6425). At 6445, the algorithmreturns.

In some embodiments, the algorithm shown in 16D provides more detailsregarding operation 6310 of FIG. 16C.

FIG. 16E describes another algorithm that accomplishes the sameobjective as the Complex Deltas algorithm with much less computing, butlower accuracy. However, in most cases this approach works well. Titled“Simple Deltas”, the algorithm simply looks at the X and Y placement ofeach touch point in a very simple sense and takes advantage of the factthat most of the time, the index finger is placed lower than the middlefinger, and so on. As shown in FIG. 16E, simple deltas algorithm (6500)includes moving left to right, calculating the location of each touchpoint (6505). Then, determining (6510) if left-most point issignificantly lower than next point to the right. If no (6510-N), thealgorithm determines that the fingers down are (3, 4, 5) (6520). If yes(6510-Y), the algorithm continues and determines that the left-mostfinger is 2. Then the algorithm checks if the next two points areroughly the same Y value. If yes (6525-Y), the algorithm determines thatthe fingers down are (2, 3, 4) (6530) and, if no (6525-N), the algorithmdetermines that the fingers down are (2, 4, 5) (6535). At 6540, thealgorithm returns.

In some embodiments, the algorithm shown in 16E provides more detailsregarding operation 6315 of FIG. 16C.

FIG. 16F describes an algorithm whereby a common Kalman filter isapplied to track fingers as they move on the touch surface (as shown foroperations 6500, 6505, 6510, 6515, and 6520). In some embodiments, thealgorithm shown in 16F provides more details regarding operation 6115 ofFIG. 16A.

FIG. 16G describes, in detail, how the system identifies and tracks theuser's fingers as they are hovering above the touch surface. Many of thesame algorithms that are used for touch data are also used for hoverdata. The result of this algorithm is assignment of locations for eachof the user's fingers based on the hover data. In some embodiments, thedetails provided with respect to 6200 (FIG. 16B) also apply for 6655 and6690 (but for hover instead of touch data), the details provided withrespect to 6300 (FIG. 16C) also apply for 6660 and 6695 (but for hoverinstead of touch data), and the details provided with respect to 6500(FIG. 16F) also apply for 6680 (but for hover instead of touch data). Insome embodiments, the pictured operations (e.g., 6600, 6605, 6610, 6615,6620, 6625, 6630, 6635, 6640, 6645, 6650, 6665, 6670, and 6675) are allused to accumulate and parse hover data so that it can be passed to andprocessed by operations 6200, 6300, and 6500.

FIG. 16H is a flowchart depicting a method 1680 of disambiguating keyactuations using finger-to-keys encoding information, in accordance withsome embodiments. In some embodiments, the method 1680 is performed byan electronic device (e.g., device 100, FIG. 1) and/or one or morecomponents of the device 100 (e.g., touch sensors 120, hover sensors122, display 130, vibrator 180, operating system 151, etc.). In someembodiments, the method 1680 is governed by instructions that are storedin a non-transitory computer-readable storage medium and that areexecuted by one or more processors of a device, such as the one or moreprocessors 110 of device 100 (FIG. 1). For ease of explanation, thefollowing describes method 1680 as performed by the device 100 (in otherembodiments or circumstances one or more of the operations of method1680 are performed by individual components of the device 100, e.g.,those pictured in FIG. 1). Some operations in method 1680 are,optionally, combined and/or the order of some operations is, optionally,changed.

As described below, the method 1680 (and associated interfaces)provide(s) a seamless way to disambiguate key actuations. Thus, usersare able to type quickly and the device 100 will accurately process eachkeystroke. In this way, method 1680 helps to ensure that users can typequickly without having to constantly correct erroneously interpretedkeystrokes. Thus, method 1680 helps reduce the cognitive burden and thetime that a user must spend to type at a virtual keyboard, therebycreating a more efficient human-machine interface. For battery-operatedelectronic devices, accurately processing keystrokes and disambiguatinginput faster and more efficiently both conserves power and increases thetime between battery charges.

The method 1680 begins when the device builds (1681) a repository ofwords typed by a user on the touch-sensitive display, each respectivecharacter of a respective word is associated with informationidentifying a finger that was used by the user to select the respectivecharacter. For example, the device 100 builds the repository during alearning process (e.g., in which the user types common phrases),retrieves words typed by a user at other devices distinct from device100 (e.g., a laptop, a tablet, etc.) and adds them to the repository, ormonitors words as they are typed (e.g., while the user is using variousapplications, such as word processing and messaging applications) andadds them to the repository.

In some embodiments, the device presents (1682), on a touch-sensitivedisplay, a virtual keyboard with a plurality of keys (e.g., as shown inFIGS. 11C and 13A). In response to detecting selection of a first key ofthe plurality of keys of the virtual keyboard, the device determines(1683) a first finger that was used by the user to select the first key.After detecting selection of at least the first key and in response todetecting an ambiguous selection of an unknown key of the plurality ofkeys of the virtual keyboard, the device determines a second finger thatwas used by the user to select the unknown key (1684). In someembodiments, the finger that was used to select the unknown key can bedetermined (e.g., using hover sensors), but the key selection itself isambiguous (e.g., because the selected is at a location bordering twodifferent keys).

The device also retrieves (1685) information associated with candidatewords from the repository of words typed by the user, the informationassociated with the candidate words including (i) a first characterassociated with the first key and information identifying the firstfinger as having selected the first key and (ii) information identifyingthe second finger as having selected a different key that is distinctfrom the first key. The device then determines (1686) that the unknownkey is a second key of the plurality of keys of the virtual keyboardbased at least in part on the information associated with the candidatewords (in other words, the device disambiguates an ambiguous keyselection by using known information about the fingers that were used toprovide respective actuations/selections of keys).

In some embodiments, the information associated with the candidate wordsincludes an association of the second finger to the second characterthat corresponds to the second key.

In some embodiments of the method 1680, the device also providesbaseline rest positions for at least eight of the user's fingers on thevirtual keyboard. For example, the device detects the baseline restpositions based on where a user's fingers come in contact with the TSDor based on stored information about the baseline rest positions.

In some embodiments of the method 1680, each respective character of arespective word is also associated with a respective first distance,relative to the baseline rest position, travelled by the finger that wasused by the user to select the respective character.

In some embodiments of the method 1680, the device eliminates one ormore of the candidate words in accordance with a determination that asecond distance, relative to the baseline rest position for the secondfinger, travelled by the second finger to select the unknown key exceedsa threshold distance as compared to a respective first distancetravelled by the second finger to select the different key. In someembodiments, if the second finger travelled too far or too little toreach the different key (relative to the distance travelled by thesecond finger to select the unknown key), then that candidate word iseliminated. In some embodiments, the threshold distance is 1 px, 1.5 px,2 px, or the like.

In some embodiments of the method 1680, the baseline rest positions aredetermined based on one of the following: contact with thetouch-sensitive display before the user selects the first key, hoverdetection when the user's fingers are within a predefined distance ofcontacting the touch-sensitive display, or baseline rest positionsretrieved from a memory of the electronic device.

In some embodiments of the method 1680, building the repository of wordsincludes building the repository of words during a training session inwhich the user types a predefined sequence of words.

In some embodiments of the method 1680, the device, while building therepository of words, identifies unique typing characteristics associatedwith the user and associating the identified unique typingcharacteristics with the repository of words (e.g., using any of thetechniques discussed herein). In some embodiments, the device istherefore able to build distinct repositories for each user of theelectronic device, resulting in improved typing accuracy and ambiguouskey selection resolution that is tailored to each user's typingcharacteristics.

In some embodiments, the method 1680 disambiguates a single selection ofthe unknown key and determines a key that the user intended to selectbased on information identifying a finger that was used to type theunknown key. In other words, the method 1680 does not necessarily detectselection of the first key (as discussed above) before being able toaccurately disambiguate the ambiguous selection of the unknown key.

FIGS. 17A and 17B are visual depictions of a proximity field captured byhover sensors, in accordance with some embodiments. In somecircumstances, the hover sensors can detect peaks and valleys associatedwith the user's finger locations (shown as points 1702 through 1705,respectively). In FIG. 17B, the middle finger is approaching the touchsurface to select a key, and its proximity field is shown morepronounced at 1711. In this way, the shape of the proximity field maychange as the user is typing. The way in which the proximity fieldchanges shape and location is directly correlated by the system to keyselection actions.

FIG. 17A also provides a visual depiction of what a touch capacitivefield might look like with a user resting the fingers of their righthand just above the surface. The slightly raised areas marked 1702through 1705 represent the user's fingers (index, middle, ring, andlittle finger respectively). Note how the area 1703 representing themiddle finger has a slightly stronger signal since the middle fingertypically is the longest and would rest closer to the surface.

FIG. 17B also provides a visual depiction of the middle finger movingcloser to the touch surface and thus results in a stronger touch signalat 1711. When a key selection is made at or near the location associatedwith the right middle finger, the hover detection data can correlatethat indeed, the key selection was made by the middle finger. Aspreviously described, knowing which finger was used to select a key cangreatly enhance the disambiguation process.

FIG. 18A and 18B are visual depictions of a proximity field in which thehover sensors do not provide detailed data as to the position of theuser's fingers. In this case, the user's hand print informationcollected and recorded during set-down events is used, along with datafrom the hand model database, to interpolate where the user's fingersare most likely located within the proximity field (shown as 801).

For example, in the case of fingers hovering above the touch surface,the touch regions may combine into a single region as shown in FIG. 18A.In this case, the system may apply knowledge of human anatomy toestimate the placement of each finger. In FIG. 18B, 1801 shows eachindividual finger estimated from the combined touch region, with thepointer finger separate slightly from the other three fingers (as istypically the case with fingers in a resting position).

Finger identification may also be performed using unambiguouskeystrokes. (This is the reverse of the key disambiguation algorithmdescribed in the next section). When a key is selected unambiguously,the system uses the finger-mapping for that key to help determine whichfinger was used, and then continues to track the touch region associatedwith that finger subsequent to the key selection.

Fingers may also be uniquely identified, for example and withoutlimitation, by: (i) the size of touch impression they leave on the touchsurface, (ii) the force with which they tap on the surface (the indexfinger can tap much harder than the little finger), (iii) theirrespective weights as they rest on the surface, (iv) which keys they areused to type on, and (iv) the relative placement of touches generated byeach finger.

In some embodiments, hand samples are collected from numerous differentages and genders to form a database that models typical hand and fingersizes, and relative distances and angles between each finger pair. Thishand model database can be used to help uniquely identify each fingerbeing sensed, and even fingers that are missing from the sensor data.FIG. 19 provides an example of a plot of data derived from 35 differenthand samples. In particular, FIG. 19 shows a sample of the data providedin the hand model database. In this plot, the distance ratio betweenfinger 2 (index finger) and finger 5 (little finger) is shown as 1901.As the plot shows, this data is nicely separated from all other distanceratios. Thus, if a distance ratio is found to be over 0.25, it can beassumed that the user has both their index and little fingers touching.Many other parameters including relative angles between each fingertouch points are also stored in the hand model database. Thisinformation is used to verify measured data as well as interpolatefinger positions when explicit measured data is not available.

FIG. 20 illustrates one benefit of knowing which finger is associatedwith discrete touch events (e.g., key actuations on an onscreenkeyboard). As an example, a user has selected between the “i” and “o”keys with finger 3 (middle finger) at 2001, then selects the letter “n”with finger 2 (index) at 2002, and finally selects a space with theirthumb at 2003. The first selection is ambiguous, since it is between thetwo keys (“i” and “o” keys). In some circumstances, it is very difficultfor language models to determine which word the user intended: “in” or“on”. Both words are very commonly used in very similar circumstances.Without any additional information, an auto-correct system will estimateand likely be wrong 50% of the time.

However, with the system of the present invention, information is storedas the user types, which forms associations between letters typed andwhat finger was used to type them (“key-to-finger” database). In thisexample, the key-to-finger database reveals that the user is aten-finger touch typists who commonly types the letter “i” with theirmiddle finger, and the letter “o” with their ring finger. The systemapplies this information and assigns the letter “i” to the firstcharacter since it was typed using the middle finger.

In some embodiments, the system explicitly determines that the user is atouch typist (e.g., based on the user's typing style), and, based onthat determination, the system maps the ambiguous touch 2001 made usingthe middle finger to the key representing the letter “i”, since that keyis typically selected using the middle finger by touch-typists.Therefore, some embodiments also provide ways to detect and determinetyping styles that are specific to particular users. Studies have shownthat roughly 30% of typists use eight, nine or ten fingers (“touchtypists”) and less than 5% use only two or three fingers. The remaining65% are hybrid typists who use 4, 5, 6, or 7 fingers. Models for each ofthese typing styles are stored in a database and the system constantlyanalyses the typing style of the user to determine what category oftypist they are. Once identified, these models can be used to determinewhich finger is used for which key, and thus contributes to thedisambiguation algorithms as described herein.

The typing style of users can vary greatly, and often becomes veryunique for each individual typist. The system described herein can learnand adapt to each user's typing style. When a key is unambiguouslyselected, the system correlates which finger was used for that selectionand stores it in a database. Multiple entries for each key may be storedto determine the most likely finger used to select that key in differentcircumstances (since different fingers may be used for the same key indifference circumstances). In this way, the system can become tuned toeach user's particular typing style and become more and more accurateover time. A user's typing style profile may be stored in a database andselected by the user manually, or automatically applied once aparticular user's typing style is identified by the system. In apreferred embodiment, the typing style “signature” of a user may bestored in the cloud (remotely stored via the internet). The system maydetect the characteristics of the user typing on a given device, comparethose characteristics with the database stored in the cloud, identifythe user, and load their specific settings (including languagedatabases, word lists, and so on).

In another embodiment, the computer processing for identifying theuser's typing style, disambiguating keystrokes, and otherprocessor-intensive tasks can be performed in the cloud. The raw touch,tap, hover, and other sensor data is transmitted to a remote processorin the cloud via a network or internet connection, where it is processedand results returned to the local device in the form of instructions,keystrokes, or other commands.

Even the same user can have varying typing styles. For example, atouch-typist may injure a finger and be unable to use that finger totype for a period of time while the finger is healing. The human body ishighly adaptable, and the user would likely change their typing style toavoid using the injured finger and quickly become adept at the newmodified typing style. So, it is important that the system is able todynamically adapt to changes in the typing style—even from the sameuser. It is able to do this by continuously tracking which finger isused to select keys and modifying the user's Typing Print accordingly.Emphasis is given in the algorithm to the more recent user actions.

Because it may take time for the system to learn a particular user'styping style, the user may speed up the process by performing anexercise to explicitly teach the system their style. In one embodiment,the user may type the phrase “The quick brown fox jumped over the lazydogs”; a phrase that contains all 26 letters of the English alphabet. Asthe user types the phrase, the system records which finger is used foreach letter. Of course, other more involved methods of calibratingexercised could be employed.

FIG. 21 is a flowchart of a method whereby the key-to-fingerassociations are derived. Note that the system works both ways: first,it stores information in the database when an unambiguous key selectiontakes place. Secondly, it pulls from the database when a key selectionis ambiguous.

In some embodiments, the system can also use unambiguous key selectionsto determine which finger was used (in the case where the fingertracking algorithm is unable to determine finger positions). So, thekey-to-finger database can be used to disambiguate both a key selectionand which finger was used, assuming one or the other is not ambiguous.

As discussed above, many typists position their fingers just above thekeys of a keyboard while typing. The ability to sense a user's fingersabove a touch surface greatly enhances the touch-typing experience in anumber of ways. First, as previously mentioned, the position of thekeyboard halves can be defined by the location of the hovering fingers.In particular, the index fingers define the location for the home-keys“F”, and “J”. Secondly, by tracking hovering fingers, the system canmake a more accurate calculation as to which finger was used to make akey selection. As a hovering finger approaches the keyboard andeventually touches it, the touch signal gets stronger.

The problem of inaccurate disambiguation is particularly acute on theqwerty keyboard layout where the vowels i, o, and u are each adjacentone to another. Words like put, pit, and pot become very difficult totell apart if the second character is input inaccurately. Theembodiments discussed herein address this problem by recordingunambiguous keystrokes when they occur and associating them to theuser's fingers. Then, when an ambiguous keystroke occurs, the system isable to easily discern from the previously stored data which key theuser really intended, based on which finger they used to select it.There are numerous statistical techniques that may be applied incombination with (or as alternatives to) the foregoing to improveaccuracy of the finger location and touch location data . What isdescribed above is a simplified version intended for explanatorypurposes, and the spirit and scope of the various embodiments disclosedherein are in no way intended to be restricted to the foregoingexamples.

As mentioned above, some embodiments allow users to determine a displayposition for an onscreen/virtual keyboard based on where they positiontheir fingers on a touch-sensitive surface. In some embodiments, adynamic onscreen keyboard system tracks where the user is typing andmoves all or part of the keyboard on the display to follow the user'sfingers (“drifting”). This is helpful on touchscreens and otherinterfaces that lack tactile feedback when typing on the keys (as thereis with mechanical keyboards). The onscreen keyboard can also be movedby the user by resting fingers on the surface and sliding hands (orrotating hands or otherwise moving fingers together) in a desireddirection (“rest and slide”). In both cases, the system limits how theonscreen keyboard is relocated so-as to prevent parts of it from movingoff the visible display area of the touch surface.

In some embodiments, there are at least three ways the proposed keyboardsystem may render a keyboard image that extends at least partly off thescreen: (1) Off-Screen Set-down: a user set-down is too far off center;(2) Rest & Slide Off-screen: the user rests and slides too far offcenter; and (3) Off-screen Drift: the keyboard drifts off-screen as theuser is typing. Each of these is discussed in turn below.

(1) Off-Screen Set-down. In some embodiments, this condition occurs whena user sets his or her fingers down on the surface in what is deemed tobe a valid set-down, but in a position that forces at least some part ofthe onscreen keyboard off the screen (i.e., in a location correspondingto a home-row position of a keyboard image that leaves insufficientscreen space on which to display the full keyboard).

Behavior Implemented by the Keyboard System. Keyboard appears at thelocation indicated by the user's hand position, even if it takes part ofthe keyboard off screen. A constant vibration is output as long as theuser's hands are resting outside the allowed area. Blue rectangular cuesare shown onscreen indicating to the user the “safe zone” where theyshould move their fingers. When the user lifts their hands after theset-down, the keyboard snaps back (via an animation) to the locationclosest matching the user's set-down region but where all parts of thekeyboard are still visible on-screen.

(2) Rest & Slide Off-Screen. In some embodiments, this condition occurswhen a user's fingers are set down on the surface in what is deemed tobe a valid set-down, but then slid to a position that takes at leastpart of the onscreen keyboard off the visible screen area. BehaviorImplemented by the Keyboard System: Keyboard should follow the user'sfingers, even to the extent that it is moved off the screen; A constantvibration is output as long as the user's hands are resting outside theallowed area; Blue rectangular cues are shown onscreen indicating to theuser the “safe zone” where they should move their fingers; and When theuser lifts their hands after the set-down, the keyboard snaps back (viaan animation) to the location closest matching the user's set-downregion but where all parts of the keyboard are still visible on-screen.

So, the behavior between conditions 1 and 2 (off-screen set down andrest & slide, respectively) is basically the same, the only differencebeing whether the set-down occurs onscreen or not. FIGS. 24A-24B show anexample of an off-screen set-down that occurs initially or a dragging ofthe keyboard that occurs as a user slides to a position that is at leastpartially off-screen (FIG. 24A). In response, the keyboard snaps back tovisible region after off-screen lift-off (FIG. 24B).

(3) Drifting Off-Screen. In some embodiments, this condition occurs whenthe keyboard starts within the visible region, but then the user driftstheir hands while typing to the point that the keyboard follows themoff-screen. Behavior Implemented by the Keyboard System: the keyboardstops drifting at the visible edges; no warning sound or vibration isplayed (since the user's hands won't necessarily be in contact with thetouch-sensitive surface); and a red outline of the keyboard tray isdisplayed for a notification interval (e.g., 0.5 sec, though longer orshorter times may be used) each time the user types at key locationsthat would cause the keyboard to drift further off-screen (if it wereallowed to do so). An example of a user drifting their fingersoff-screen with their right hand while typing is pictured in FIG. 24C(as shown visual feedback is provided to indicate that the user shouldadjust position back on-screen).

In another embodiment, the keyboard system compresses to fit all of thekeys of the keyboard on the visible area of the display if it is movedor drifts to an edge in a way that would otherwise force some or all ofthe keyboard off the screen. The system may implement the same effect ifthe two halves of the keyboard collide in the middle. FIGS. 23A-23E showthe “accordion” concept.

In some embodiments, the system can notify the user in a number ofdifferent ways when a collision or off-screen violation is occurring,including visual, audible, and vibrating alerts. The visual cues caninclude, for example and without limitation, changing the color of thefill and/or line of violating keys, changing the background of thekeyboard, or flashing a highly visible line down the edge of the screenon which the violation is taking place.

FIG. 22 is a flowchart depicting a method 2200 of adaptively presentinga virtual keyboard, in accordance with some embodiments. In someembodiments, the method 2200 is performed by an electronic device (e.g.,device 100, FIG. 1) and/or one or more components of the device 100(e.g., touch sensors 120, hover sensors 122, display 130, vibrator 180,operating system 151, etc.). In some embodiments, the method 2200 isgoverned by instructions that are stored in a non-transitorycomputer-readable storage medium and that are executed by one or moreprocessors of a device, such as the one or more processors 110 of device100 (FIG. 1). For ease of explanation, the following describes method2200 as performed by the device 100 (in other embodiments orcircumstances one or more of the operations of method 2200 are performedby individual components of the device 100, e.g., those pictured in FIG.1). Some operations in method 2200 are, optionally, combined and/or theorder of some operations is, optionally, changed.

The method 2200 begins when the device monitors (2202) typing inputsfrom a user at a virtual keyboard that is displayed on a touch-sensitivedisplay of the device. In some embodiments the typing inputs arecontacts with a keyboard that is displayed on the touch-sensitivedisplay while, in other embodiments, the typing inputs are receivedwhile no keyboard is displayed on the touch-sensitive display.

The device determines, while monitoring the typing inputs, if the typinginputs are converging towards a midpoint of the touch-sensitive display(2204). In accordance with a determination that the typing inputs areconverging towards the midpoint of the touch-sensitive display(2204-Yes), the device provides a first feedback to the user to indicatethat the typing inputs are converging. In some embodiments, the firstfeedback includes an accordion effect that shrinks size of a middlecolumn of keys on the virtual keyboard (as pictured in the sequence ofexample virtual keyboards shown in FIGS. 23A-23E).

The device also determines, while monitoring the typing inputs, if thetyping inputs are diverging away from the midpoint of thetouch-sensitive display (2206, in some instances operation 2206 isperformed after determining that typing inputs are not converging at2204-No). In accordance with a determination that the typing inputs arediverging away from the midpoint of the touch-sensitive display(2206-Yes), the device provides a second feedback to the user toindicate that the typing inputs are diverging (as pictured in FIGS.24A-24C).

In some embodiments, operations 2204 and 2206 are performed together,for example, by having the device determine whether the typing inputsare converging towards a midpoint of the touch-sensitive display ordiverging away from the midpoint of the touch-sensitive display. In someembodiments, determining whether the typing inputs are converging ordiverging includes comparing a location on the touch-sensitive displayassociated with a first typing input with a display position on thetouch-sensitive display associated with a key that corresponds to thefirst typing input.

In some embodiments, the device 100 determines whether the typing inputssatisfy keyboard-modification criteria. In some embodiments, thekeyboard-modification criteria include a first criterion that issatisfied when the typing inputs are determined to be converging towardsa midpoint of the touch-sensitive display. In some embodiments, thekeyboard-modification criteria include a second criterion that issatisfied when the typing inputs are determined to be diverging awayfrom the midpoint of the touch-sensitive display. In some embodiments,the keyboard-modification criteria include only the first or the secondcriterion. In some embodiments, the keyboard-modification criteriainclude both of the first and the second criterion. In embodiments inwhich the keyboard-modification criteria include only the firstcriterion, the method 2200 includes operations 2202, 2204, 2210, 2208,and 2214 (FIG. 22). In embodiments in which the keyboard-modificationcriteria include only the second criterion, the method 2200 includesoperations 2202, 2206, 2212, 2208, and 2214 (FIG. 22).

In some embodiments of the method 2200, the typing inputs are receivedat a virtual keyboard with a plurality of keys displayed at predefinedlocations on the touch-sensitive display of the electronic device.

In some embodiments of the method 2200, providing the first feedbackincludes displaying at least one key of the plurality of keys at a newlocation closer to the midpoint of the touch-sensitive display relativeto a respective predefined location associated with the at least one keyof the plurality of keys, the respective predefined location beingwithin a threshold distance of the midpoint of the touch-sensitivedisplay. As pictured in FIG. 23B, because the typing inputs aredetermined to be converging towards the midpoint, the location of the“G” key is shifted towards the midpoint (as compared to its originalposition shown in FIG. 23A).

In some embodiments of the method 2200, providing the first feedbackincludes displaying each key of the plurality of keys at a respectivenew location closer to the midpoint of the touch-sensitive displayrelative to a respective predefined location associated with each key.As pictured in FIG. 23B, because the typing inputs are determined to beconverging towards the midpoint, the location of each key in the middlecolumn (e.g., “G” key, “T” key, “B” key, “Y” key, “H” key, and “N” key)is shifted towards the midpoint (as compared to their original positionsshown in FIG. 23A).

In some embodiments of the method 2200, providing the first feedbackincludes reducing a display size associated with each key of theplurality of keys.

In some embodiments of the method 2200, providing the first feedbackincludes reducing a display size associated with at least one key of theplurality of keys, the at least one key being displayed at a respectivepredefined location that is within a threshold distance of the midpoint(e.g., 1 px, 2 px, 3 px, etc.) of the touch-sensitive display. Aspictured in FIG. 23B, because the typing inputs are determined to beconverging towards the midpoint, the display size associated with eachkey that is within the threshold distance (e.g., “G” key, “T” key, “B”key, “Y” key, “H” key, and “N” key) is reduced (as compared to theirdisplay sizes shown in FIG. 23A). In some embodiments, providing thefirst feedback includes reducing a display size associated with each keyof the plurality of keys that is displayed at a respective predefinedlocation that is within the threshold distance.

In some embodiments of the method 2200, the method includes: afterproviding the first feedback, in accordance with a new determinationthat the typing inputs are still converging towards the midpoint,reducing a display size associated with the at least one key. Aspictured in FIG. 23C, because the typing inputs are determined to stillbe converging towards the midpoint, the display size of the “G” key isreduced (as compared to its display size shown in FIG. 23B).

In some embodiments, instead of or in addition to reducing display sizesfor keys, the device displays a keyboard with overlapping positions forrespective keys on the keyboard. For example, as shown in FIG. 24D,positions for the t and y keys, the g and h keys, and the b and n keysare displayed as overlapping. In other words, at least two keys (e.g.,the t and the y key) occupy at least a portion of the same display spaceon the keyboard. If an input is detected (e.g., a tap) occurs at thatoverlapping same display space, then disambiguation is performed todetermine which key the user intended to select. In some embodiments,both letters (i.e., letters displayed on the respective keys that occupythe overlapping same display space) are considered as valid keycandidates and the word database/language model can be used to providedisambiguation information. In some embodiments, determining whichfinger was used to provide the ambiguous input (in addition to fingersthat were used to type previous inputs) may be used to help determinewhich key the user intended to select.

In some embodiments of the method 2200, the method includes: inaccordance with a determination that the typing inputs have not beenreceived for a predetermined period of time, re-displaying the keys atthe predefined locations (e.g., returning to the display shown in FIG.23A).

In some embodiments of the method 2200, the method includes: inaccordance with a determination that the typing inputs are no longerconverging towards the midpoint of the touch-sensitive display,re-displaying the keys at the predefined locations (e.g., returning toeach key to its predefined location as shown in FIG. 23A).

In some embodiments of the method 2200, providing the second feedbackincludes displaying a predefined graphic at the virtual keyboard for apredefined period of time. For example, a red outline is displayedaround the virtual keyboard for 0.5, 1, 1.5, or 2 seconds (as shown inFIG. 24C). In some embodiments, providing the second feedback includesproviding tactile feedback to the user (e.g., causing the electronicdevice to vibrate).

In some embodiments of the method 2200, providing the second feedbackincludes displaying at least two keys of the plurality of keys at newlocations that are closer to a first edge or a second edge of thetouch-sensitive display relative to respective predefined locations ofthe at least two keys, the respective predefined location for a firstkey of the at least two keys being within a threshold distance of thefirst edge of the touch-sensitive display and the respective predefinedlocation for a second key of the at least two keys being within thethreshold distance of the second edge of the touch-sensitive display. Insome embodiments, providing the second feedback includes displaying eachkey of the plurality of keys at a respective new location that is closerto a first edge or a second edge of the touch-sensitive display. Inother words, each half of the keyboard is shifted towards an edge of thetouch-sensitive display.

In some embodiments of the method 2200, providing the second feedbackincludes reducing a display size associated with at least two keys ofthe plurality of keys, a first key of the at least two keys beingdisplayed at a respective predefined location that is within a thresholddistance of a first edge of the touch-sensitive display and a second keyof the at least two keys being displayed at a location that is withinthe threshold display of a second edge, opposite the first edge, of thetouch-sensitive display. In some embodiments, providing the secondfeedback includes reducing a display size associated with each key ofthe plurality of keys.

In some embodiments of the method 2200, the method includes: inaccordance with a determination that the typing inputs have not beenreceived for a predetermined period of time, re-displaying the keys atthe predefined locations.

In some embodiments of the method 2200, the method includes: inaccordance with a determination that the typing inputs are no longerdiverging away from the midpoint of the touch-sensitive display,re-displaying the keys at the predefined locations. The variousstructures, methods and techniques disclosed herein may be implementedthrough execution of one or more sequences of instructions (i.e.,software program(s)) within a computer system, or by a custom-builthardware ASIC (application-specific integrated circuit), or programmedon a programmable hardware device such as an FPGA (field-programmablegate array), or any combination thereof within or external to thecomputer system.

Any of the various methodologies disclosed herein and/or user interfacesfor configuring and managing same may be implemented by machineexecution of one or more sequences instructions (including related datanecessary for proper instruction execution). Such instructions may berecorded on one or more computer-readable media for later retrieval andexecution within one or more processors of a special purpose or generalpurpose computer system or consumer electronic device or appliance, suchas the system, device or appliance described in reference to FIG. 12.Computer-readable media in which such instructions and data may beembodied include, but are not limited to, non-volatile storage media invarious forms (e.g., optical, magnetic or semiconductor storage media)and carrier waves that may be used to transfer such instructions anddata through wireless, optical, or wired signaling media or anycombination thereof. Examples of transfers of such instructions and databy carrier waves include, but are not limited to, transfers (uploads,downloads, e-mail, etc.) over the Internet and/or other computernetworks via one or more data transfer protocols (e.g., HTTP, FTP, SMTP,etc.).

In the foregoing description and in the accompanying drawings, specificterminology and drawing symbols have been set forth to provide athorough understanding of the disclosed embodiments. In some instances,the terminology and symbols may imply specific details that are notrequired to practice those embodiments. For example, any of the specificdimensions, form factors, signal path widths, signaling or operatingfrequencies, component circuits or devices and the like can be differentfrom those described above in alternative embodiments. Additionally,links or other interconnection between system components or internalcircuit elements or blocks may be shown as buses or as single signallines. Each of the buses can alternatively be a single signal line, andeach of the single signal lines can alternatively be buses. Signals andsignaling links, however shown or described, can be single-ended ordifferential. A signal driving circuit is said to “output” a signal to asignal receiving circuit when the signal driving circuit asserts (orde-asserts, if explicitly stated or indicated by context) the signal ona signal line coupled between the signal driving and signal receivingcircuits. The term “coupled” is used herein to express a directconnection as well as a connection through one or more interveningcircuits or structures. Device “programming” can include, for exampleand without limitation, loading a control value into a register or otherstorage circuit within the integrated circuit device in response to ahost instruction (and thus controlling an operational aspect of thedevice and/or establishing a device configuration) or through a one-timeprogramming operation (e.g., blowing fuses within a configurationcircuit during device production), and/or connecting one or moreselected pins or other contact structures of the device to referencevoltage lines (also referred to as strapping) to establish a particulardevice configuration or operation aspect of the device. The terms“exemplary” and “embodiment” are used to express an example, not apreference or requirement. Also, the terms “may” and “can” are usedinterchangeably to denote optional (permissible) subject matter. Theabsence of either term should not be construed as meaning that a givenfeature or technique is required.

Various modifications and changes can be made to the embodimentspresented herein without departing from the broader spirit and scope ofthe disclosure. For example, features or aspects of any of theembodiments can be applied in combination with any other of theembodiments or in place of counterpart features or aspects thereof.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages which are not order dependent may bereordered and other stages may be combined or broken out. Furthermore,in some embodiments, some stages may be performed in parallel and/orsimultaneously with other stages (e.g., operations 315, 320, and 325 inFIG. 3 may all be performed together or substantially in parallel).While some reordering or other groupings are specifically mentioned,others will be apparent to those of ordinary skill in the art, so theordering and groupings presented herein are not an exhaustive list ofalternatives. Moreover, it should be recognized that the stages could beimplemented in hardware, firmware, software, or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the embodiments to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples and their practical applications, to thereby enable othersskilled in the art to best utilize the embodiments and variousembodiments with various modifications as are suited to the particularuse contemplated.

What is claimed is:
 1. A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of an electronic device with a touch-sensitive display, the one or more programs comprising instructions for: monitoring typing inputs received from a user at the touch-sensitive display; determining whether the typing inputs are converging towards a midpoint of the touch-sensitive display or diverging away from the midpoint of the touch-sensitive display; in accordance with a determination that the typing inputs are converging towards the midpoint of the touch-sensitive display, providing a first feedback to the user to indicate that the typing inputs are converging; and in accordance with a determination that the typing inputs are diverging away from the midpoint of the touch-sensitive display, providing a second feedback to the user to indicate that the typing inputs are diverging.
 2. The non-transitory computer-readable storage medium of claim 1, wherein the typing inputs are received at a virtual keyboard with a plurality of keys displayed at predefined locations on the touch-sensitive display of the electronic device.
 3. The non-transitory computer-readable storage medium of claim 2, wherein the instructions for providing the first feedback comprise instructions for displaying at least one key of the plurality of keys at a new location closer to the midpoint of the touch-sensitive display relative to a respective predefined location associated with the at least one key of the plurality of keys, the respective predefined location being within a threshold distance of the midpoint of the touch-sensitive display.
 4. The non-transitory computer-readable storage medium of claim 2, wherein the instructions for providing the first feedback comprise instructions for reducing a display size associated with each key of the plurality of keys.
 5. The non-transitory computer-readable storage medium of claim 2, wherein the instructions for providing the first feedback comprise instructions for reducing a display size associated with at least one key of the plurality of keys, the at least one key being displayed at a respective predefined location that is within a threshold distance of the midpoint of the touch-sensitive display.
 6. The non-transitory computer-readable storage medium of claim 5, wherein the one or more programs include instructions for: after providing the first feedback, in accordance with a new determination that the typing inputs are still converging towards the midpoint, reducing a display size associated with the at least one key.
 7. The non-transitory computer-readable storage medium of claim 3, wherein the one or more programs include instructions for: in accordance with a determination that the typing inputs have not been received for a predetermined period of time, re-displaying the keys at the predefined locations.
 8. The non-transitory computer-readable storage medium of claim 3, wherein the one or more programs include instructions for: in accordance with a determination that the typing inputs are no longer converging towards the midpoint of the touch-sensitive display, re-displaying the keys at the predefined locations.
 9. The non-transitory computer-readable storage medium of claim 2, wherein the instructions for providing the second feedback comprise instructions for displaying a predefined graphic at the virtual keyboard for a predefined period of time.
 10. The non-transitory computer-readable storage medium of claim 2, wherein the instructions for providing the second feedback comprise instructions for displaying at least two keys of the plurality of keys at new locations that are closer to a first edge or a second edge of the touch-sensitive display relative to respective predefined locations of the at least two keys, the respective predefined location for a first key of the at least two keys being within a threshold distance of the first edge of the touch-sensitive display and the respective predefined location for a second key of the at least two keys being within the threshold distance of the second edge of the touch-sensitive display.
 11. The non-transitory computer-readable storage medium of claim 2, wherein the instructions for providing the second feedback comprise instructions for reducing a display size associated with at least two keys of the plurality of keys, a first key of the at least two keys being displayed at a respective predefined location that is within a threshold distance of a first edge of the touch-sensitive display and a second key of the at least two keys being displayed at a location that is within the threshold display of a second edge, opposite the first edge, of the touch-sensitive display.
 12. The non-transitory computer-readable storage medium of claim 10, wherein the one or more programs include instructions for: in accordance with a determination that the typing inputs have not been received for a predetermined period of time, re-displaying the keys at the predefined locations.
 13. The non-transitory computer-readable storage medium of claim 10, wherein the one or more programs include instructions for: in accordance with a determination that the typing inputs are no longer diverging away from the midpoint of the touch-sensitive display, re-displaying the keys at the predefined locations.
 14. A method, comprising: at an electronic device with a touch-sensitive display: monitoring typing inputs received from a user at the touch-sensitive display; determining whether the typing inputs are converging towards a midpoint of the touch-sensitive display or diverging away from the midpoint of the touch-sensitive display; in accordance with a determination that the typing inputs are converging towards the midpoint of the touch-sensitive display, providing a first feedback to the user to indicate that the typing inputs are converging; and in accordance with a determination that the typing inputs are diverging away from the midpoint of the touch-sensitive display, providing a second feedback to the user to indicate that the typing inputs are diverging.
 15. An electronic device, comprising: a touch-sensitive display; one or more processors; and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to: monitor typing inputs received from a user at the touch-sensitive display; determine whether the typing inputs are converging towards a midpoint of the touch-sensitive display or diverging away from the midpoint of the touch-sensitive display; in accordance with a determination that the typing inputs are converging towards the midpoint of the touch-sensitive display, provide a first feedback to the user to indicate that the typing inputs are converging; and in accordance with a determination that the typing inputs are diverging away from the midpoint of the touch-sensitive display, provide a second feedback to the user to indicate that the typing inputs are diverging. 